Techniques of Data Warehousing

Techniques of Data Warehousing Data Warehousing is the method for reporting and data analysis, also known to be the care component of business intelligence environment.   Data can be a wide range of things, form financial to management.   With everybody within the organization having access to some form of data; security and integrity is always at risk. A Transactional Database is where a database transaction might consist of one or more data-manipulation statements and queries, each reading and/or writing information in the database.   Ex. Gym memberships, credit card purchases and mostly every banking transaction in all countries are recorded in databases unless youre paying cash.   The risk and integrity of these transactions are always accessed.   Many insurance company take the risk that their customers may never need them but always pay the bill. Even though there are some laws regarding the disclosure of health and other private information.   But the legal protection of privacy regarding the disclosure of grocery shopping habits and other things for example is slim to none in the US.   Therefore, you are at the mercy of the self-imposed privacy policies of the individual companies you deal with along with your ability to stay out of those transactional databases in the first place. Within a data warehouse you have two systems in place; OLTP and OLAP.   OLTP (On-line Transaction Processing) is characterized by a large number of short on-line transactions (INSERT, UPDATE and DELETE). The main emphasis for OLTP systems is put on very fast query processing, maintaining data integrity in multi-access environments and an effectiveness measured by number of transactions per second.   OLAP (On-line Analytical Processing) is characterized by relatively low volume of transactions. Queries are often very complex and involve aggregations. For OLAP systems a response time is an effectiveness measure. OLAP applications are widely used by Data Mining techniques. Another important factor to consider is the use of Business Intelligence.   Business Intelligence or BI is the technology infrastructure for gaining maximum information from available data for the purpose of improving business processes.   Typical BI infrastructure components are as follows: software solution for gathering, cleansing, integrating, analyzing and sharing data.   Business Intelligence produces analysis and provides believable information to help making effective and high quality business decisions. Data across Borders have become more common and frequent in everyday business.   Over the last 20 years, patterns of global dataflow have evolved at a rapid pace due to developments in global communication networks and business processes. As data is moved from data center to data center and/or across borders, security breaches become a tangible risk. To effectively protect data you must consider its lifecycle. The main features of the data lifecycle are: Create/Capture: To Receive or create data, whether captured from a website, a file transfer or a physical acquisition, will affect handling. Every method of creation or capture is going to require a different form of protection to ensure the information is safeguarded. Index and Classify: Once the data has been securely acquired, certain rules must be applied. The first step is to identify the type of data acquired. Is it personally identifiable information (PII)? Is it an image or a document? What kind of document is it?   Categorizing the document will make the process mare efficient. Store/Manage:   Where the data is stored will drive what protection controls are applied. If the data consists of PII or potential PII, then the organization may be legally required to store the data in a disk-based encryption format and encrypt backup copies of the data. Retrieve/Publish: After securely transferring data across the border, enable availability for use by ensuring that data is encrypted at each stage when transferred, stored and displayed.   Data cannot be decrypted in countries where it is not being transferred to, and access to systems such as network paths which enable cross-border transfers must be controlled. Process: To ensure the data is only used for authorized purposes and in compliance with applicable laws, application controls and metadata tagging are helpful tools. Archive: Once Data in not nedded, issues of long-term storage in compliance with the applicable policies and legal requirements arise. Is the backup onsite or offsite? Do your backups cross international borders? Are the backups governed by other countries privacy and data protection laws? The answers to these questions will help ensure that all potential risk areas are mitigated. Destroy: Sooner or later data will be deemed unusable, in accordance with applicable legislation. Ensure the destruction of archives, files, physical copies and any other copies. However, processes need to be in place for data excluded from regularly scheduled destruction cycles. For example, data subject to legal holds and discovery requests, as well as data governed by cross-border privacy legislation. Even with the most robust policies, processes and systems, continuous vigilance is required. Organizations should; Monitor change to regulatory and security. References: http://www.howtovanish.com/2009/11/transactional-databases-what-me-worry/ http://datawarehouse4u.info/OLTP-vs-OLAP.html http://datawarehouse4u.info/What-is-Business-Intelligence.html http://www.globallegalpost.com/commentary/data-across-borders-96787229/