The solution we provided involved:

• Regular client workshops;
• Frequent prototyping;
• AWS infrastructure setup and hardening;
• Detailed architecture and design work including with third party organizations;
• Data processing and analysis;
• Custom development;
• Deployment;
• Ongoing testing and support.

We produce the end-to-end solution in approximately 12 weeks of elapsed effort.

One of our major success factors was that we spent a great deal of time working closely with the principal business analyst to obtain a deep knowledge of their business model, goals & objectives, and data schema.

Once we had that shared understanding, we identified their desired reports and processes using mock reports. Instead of working with their raw data formats, we  mocked up the "desired data schema" that would facilitate their ongoing reporting. We then worked backwards to map their actual raw data with their desired state.

By keeping the myriad of complex data and technology issues "under the hood", and optimizing the solution performance for reading (reporting) instead of writing (data intake), we thus ensured that the end user reporting and analysis experience was never compromised, regardless of the state of their raw data.

Another major success factor was our decision to use a variety of best-of-breed server and serverless technologies including AWS S3, Lambda, Redshift, EC2, Tableau Server, and Tableau Desktop. These ensured our solution contained the most supported and feature-rich big data technology.

Where required, we developed code and scheduled PowerShell scripts as "glue" to integrate the various components. But we limited the custom coding and attempted to use industry standards such as SQL wherever possible, to support future upgrades.

Finally, we acknowledged up front that business queries will change over time (as they have). To accommodate this, we designed the system to handle a wide variety of future data sets and foreseeable reporting needs. In particular, we built our data warehouse using the radical MAD schema approach (see "Going MAD with Tableau and AWS Redshift").