Bio

As a frontend and backend developer who is very interested in business intelligence, I am well focused on Microsoft technologies for more than 20 years. This makes me quiet experienced in Dotnet, React, and Typescript.
As a paid worker, I work in the financial sector in Luxembourg.

Project

Summary

Before any explanation, let's go straight to the point with a bit of arrogance: the goal is to provide the best ETL engine a dotnet developer can face today

How can it be the best whereas large highly skillful teams in Microsoft, Oracle and many others are working on it since decades?

Here is how...

Context

Years after years, contexts I faced led me to the conclusion that ETL engines (even SSIS that is that is still one of the bests 15 years after its initial release) should now reach a next generation.

😲 SSIS for instance, misses so many out of the box features that makes a lot of developers very frustrated. Just to mention one typical example among many I could enumerate: the terrible absence of an efficient and fast upsert.

😕 As most of ETL engines are nearly exclusively focused on performance, I noticed that in real life, they are mostly used to integrate relatively small volumes of data. This reveals a wrong alignment between offer and demand. Most developers need to process between thousands and hundreds thousands rows, but editors keep on advertising how good they are in the integration of billions rows! At the end of the day, the sad fact is... indeed, ETL engines are monsters of performance when it is about to import billions of records... but for real life usual use cases, it is too heavy and very unproductive to develop with it. Not even to mention their integration in an application architecture.

😏 Here, many BI specialists would answer the following:

BI is a discipline of IT that is different, with a specific architecture that must be followed to get the best results. Ralph Kimball or Bill Inmon approaches shall be studied. Your vision is too much biased by your development approach.

😔 My personal belief is that the world changed; now it is this vision that is biased by an old fashioned BI approach. Computers are, like always, way more powerful than before: extracting, transforming and loading hundreds thousands rows of data is now a problem for nearly nobody anymore. Nowadays, the main problem is this one:

Can we implement with an acceptable effort an efficient, maintainable and complex ETL process in any simple architecture?

💡 Etl.Net Is meant to solve this problem.
For BI developers, it will look like... a simple Dotnet library. Nothing to do with heavy weighed tools with visual designers!

Yes...
Indeed...
So what?

Etl.Net embeds nearly all usual ETL engine features, and offers way more to reach this next generation that I refer to at the beginning of this presentation.

Funding

What for?

This project took nearly 10 months of full time development.
The version that is now accessible on nuget and github is a first draft version of it that was still, downloaded nearly 30.000 times.
The release version is 95% done, and already in production, powering the heart of the financial solution of a fintech.

Why do you need any fund as it is nearly done and working?

Simple.

• Such a tool deserves a proper documentation. Documenting is lot of time. You know the following.
• Such a tool needs to be safe. Any amendments in its code may put at risk anybody who relies on the fact it always behaves the same than currently. Therefore a set of unit tests must be completed. Lot of time as well...
• Last but not least: indeed, no visual designing tool... at the moment! A visual editor that you can embed easily in your web application will start to be developed within months.

How it will be used

Most of what exists has been done on my free time during these last 2 years. It is a daunting duty. My personal situation can hardly permit me to keep on in a fast pace.

I can keep on this way, and I give presents and holidays to my family to compensate a part of my absence. If funding is quiet important, I can stop working for couple of months to implement what remains.