it would be a very good thing for AI to replace people who are calling themselves "Data Engineers". No man who calls himself a "Data Engineer" has earned the right to use the title "Engineer". "Engineer" has meaning. It is a title to be earned, not a title to be adopted by men who are not willing to earn it.
In terms of ETL and data warehousing?
I already invented the future of ETL in 2002 when I invented "typeless, codeless, mappingless" ETL. The enabled us to build much more reliable ETL. In 2009 we moved from using C++ as the software to run ETL to using generated SQL. The rate was 1000 fields mapped in a 220 hour work month. That rate stood from 1996 to 2017, more than 20 years. In 2017 I upped it to 6-8K fields mapped per 220 hour work month. And two years ago I upped that again to 12-15K field mapped per work month. So AIs for ETL development don't help people using my software. We are already much faster than any AI can generate code and my code is more reliable anyway.
Once I got to those productivity figures I changed focus a little to the time and effort of having multiple ETL systems and checked into the ability to have "mega models" implemented even on cheap databases like SQL Server SE. Turned out it's possible. My mega models I mean one data warehouse with one data model housing many customers data from a common large operational system like an ERP or telco billing system.
So with these two inventions data warehousing development costs will be reduced by 5-6X.
This is all free and published. You and anyone else can get my data models and software for free on the link in my BIO. For those men who are the smartest men in our sector? They are very likely to make a lot of money if they build a mega model data warehouse for a large operational system they are very familiar with. That's what my colleagues and I are doing, we shall see how it works out.
AI only does things that are so easy that they were very easily automated a long time ago by those of us who know what we are doing.
just so you know. I was recently in a meeting with the CEO of a small company doing an ERP implementation. Because we were similar ages she asked me why it was that IT people were so arrogant since they were also so ignorant.
She actually said: "The average IT person does not know a debit from a credit."
And she would be right.
I can assure you that IT people, self described "data engineers" included, have zero respect on the business side of the house. Not even in a small company are IT people respected by the CEO.
I did my best to teach IT people in the 80s that if they wanted to be listened to on the business side of the house they had to EARN the respect of the business side of the house by the quality of their work.
I lost that argument in the 80s. And I was at IBM at the time.
As a result I decided to leave software development as a profession in 1989. It was clear to me by the end of 1989 that software development would never be a respected profession. The position I resigned was as the youngest ever system architect in IBM Australia at just the age of 25. I was SA for a 400 work month USD4M project to be implemented into 26 countries.....at 25.
In 1990 I transitioned to IBM Marketing. My job was to improve the productivity of IT shops using IBM hardware and software in the Insurance sector in Sydney Australia. It took a full year to go through the transition before I was allowed to speak to a customer. And I was ALREADY 8 years in IT and ALREADY the youngest system architect in the country. It didn't matter. I had to go through Marketing School and SE School like all the graduates. There was no short cut for me.
As it happened one of my first IBM customers wanted to implement Ralph Kimballs Data Interpretation System Software. I was assigned the job of implementing it in April 1991. The business results of using the data to support the management decision making process were so astonishing to us all that I changed focus from operational systems to what later became known as data warehousing.
A data warehouse is the single highest ROI investment a large company can make in the IT area. Indeed, in any area. Some of my customers doubled their profit using the ideas I gave them.
In 1991 I independently re-invented Ralphs idea of multi-level summaries in what would now be called "One Big Table". In 1992 I did a tour of all the big IBM DB2 customers who would accept my offer of a presentation of my new idea. I also spoke at all the DB2 user conferences on behalf of IBM presenting this idea.
Over the next few years my idea became widely adopted in Australia. I did some meetings with the National Australia Bank in 1992 and taught them about this idea which they then adopted. In the 1992-1997 period the NAB went from 1 billion dollars profit per year to 2 billion dollars profit per year and that was only possible with me giving them my ideas.
In 1993 I went to the Metaphor Users Conference in San Francisco and learned about the whole idea of dimensional models. Companies like Coca Cola and Proctor and Gamble were running Ralphs multi-level dimensional models.
In 1994-5 my customer did a pure research project to learn how to build dimensional models. That is where I learned to build dimensional models. That customer never went ahead with the implementation due to politics.
By 1996 I was launching the Hitachi Data Systems Asia Pacific data warehousing practice. We won the biggest data warehouse deal in the country in 1997, that being the Australian Customs Service. I was the man who introduced dimensional modeling to Australia and made it widely popular.
By 1996 it was obvious to me that data warehousing was the single biggest ROI investment a company could make and I was determined to be at the leading edge of that industry.
Bill, Ralph, Sean Kelly and I did our very best to promote honesty and integrity and not have the same happen to data warehousing as happened to software development.
We lost that argument too.
By 2002 it was clear that the area of data warehousing was going to go the same way as the area of software development. Even Bill has recently written that data warehousing has a bad reputation because so many men sold and delivered what they called a "data warehouse" that wasn't.
The IT industry overall has a very poor reputation. Indeed the Royal Bank Of Scotland had an outage where customers could not use their cards for payments for THREE DAYS. There was the recent event where literally planes were grounded because of a windows update failure.
The quality of software and the quality of data is terrible in large enterprises.
And now with the advent of AI this is being exposed. All over the internet there are people posting about how the data in companies is "not ready for AI" meaning that the data warehouses are so bad that if you put AIs on the top of them they are very likely to say things that are not true because of the very poor quality of the data.
One other thing I see very often is this.
And you can just go ask Ben at "Seattle Data Guy". Men who are self described "data engineers" and "data analysts" constantly publicly complain that they spend weeks putting together what they call "business cases" that are not only for the business case not to be accepted.
They all ask: "what can we do to have our business cases more commonly accepted?"
Only problem is they refuse to take advice from me and I was THE most successful data warehouse salesmen in Australia for quite a few years.
MEN on the IT side of the house want to have the people, both men and women, on the business side of the house buy their proposals without even being honest, let alone good at their jobs.
Their failure to be able to persuade the business side of the house to invest in their proposals, so called "business cases", is exactly because they are so dishonest they try and use unearned titles to gain credibility rather than hard work and experience.
The whole "IT" segment is in very poor shape with software crashes and bad data so common it is accepted as normal. And even now the IT segment is hyping up AI like it hyped up Hadoop in 2010 like it hyped up the internet in 1999.
And even today, the men in the IT segment will not take the advice of "honesty is the best policy".
Indeed, in case you haven't noticed, the idea that it's better to lie to people to gain money and influence had become very obvious in areas such as politics and big business. The idea of "honesty is the best policy" was long ago abandoned in favour of "if only you can fake honesty and sincerity you can make the sale".
The western world reflects the very low level of honesty in all areas of society now. Social trust is collapsing and social cohesion is collapsing and that bodes very poorly for the future of the west.
All because honesty and integrity were abandoned long ago.
Hi Andreas, it is not an “attack” to speak the truth. If men who call themselves “data engineers” wish to be respected by the men who are on the business side of they house they would have taken my advice.
That you falsley call good advice an “attack” is exactly why men like you are not respected on the business side of the house.
I have been in IT 44 years now. It is right and proper that any man who uses the title “engineer” who has not earned it be criticised for doing so.
I was an IBM Systems Engineer in the early 90s. I can assure you that it was VERY HARD to get through the IBM school to be called an associate SYSTEMS ENGINEER and VERY HARD to earn my promotion to the full level.
If men wish to bestow on themselves the title of “Data Enigineer” then they should be willing to defend themselves from my criticism and prove they earned the title. You just did the exact opposite. You played the “victim of an attack” like women do. You didn't defend the use of the title you use by demonstrating how you earned it.
Remember, in Roman times bridge builders stood beneath the bridge when it was stress tested to the full weight the bridge builder said it could carry. A real engineer often stakes his life and his reputation on the quality of his work. Even more often, he stakes other peoples lives on the quality of his work. Self described “data engineers” do not even allow peer review of their work because they know it is so bad.
As I said. If self described “data engineers” wished to be professionally respected by the business side of the house they would take my advice. I certainly have zero respect for men who call themselves data engineers and I wrote a relational database manager in 1985 as my university project.
I have been a global thought leader in data warehousing since the late 90s. And yes, I was also a system architect for operational systems in the 80s.
Hello Andreas,
it would be a very good thing for AI to replace people who are calling themselves "Data Engineers". No man who calls himself a "Data Engineer" has earned the right to use the title "Engineer". "Engineer" has meaning. It is a title to be earned, not a title to be adopted by men who are not willing to earn it.
In terms of ETL and data warehousing?
I already invented the future of ETL in 2002 when I invented "typeless, codeless, mappingless" ETL. The enabled us to build much more reliable ETL. In 2009 we moved from using C++ as the software to run ETL to using generated SQL. The rate was 1000 fields mapped in a 220 hour work month. That rate stood from 1996 to 2017, more than 20 years. In 2017 I upped it to 6-8K fields mapped per 220 hour work month. And two years ago I upped that again to 12-15K field mapped per work month. So AIs for ETL development don't help people using my software. We are already much faster than any AI can generate code and my code is more reliable anyway.
Once I got to those productivity figures I changed focus a little to the time and effort of having multiple ETL systems and checked into the ability to have "mega models" implemented even on cheap databases like SQL Server SE. Turned out it's possible. My mega models I mean one data warehouse with one data model housing many customers data from a common large operational system like an ERP or telco billing system.
So with these two inventions data warehousing development costs will be reduced by 5-6X.
This is all free and published. You and anyone else can get my data models and software for free on the link in my BIO. For those men who are the smartest men in our sector? They are very likely to make a lot of money if they build a mega model data warehouse for a large operational system they are very familiar with. That's what my colleagues and I are doing, we shall see how it works out.
AI only does things that are so easy that they were very easily automated a long time ago by those of us who know what we are doing.
Not sure why you are attacking people who are Data Engineers here? Also warehousing is just a small part of what Data Engineers do
Hi Andreas,
just so you know. I was recently in a meeting with the CEO of a small company doing an ERP implementation. Because we were similar ages she asked me why it was that IT people were so arrogant since they were also so ignorant.
She actually said: "The average IT person does not know a debit from a credit."
And she would be right.
I can assure you that IT people, self described "data engineers" included, have zero respect on the business side of the house. Not even in a small company are IT people respected by the CEO.
I did my best to teach IT people in the 80s that if they wanted to be listened to on the business side of the house they had to EARN the respect of the business side of the house by the quality of their work.
I lost that argument in the 80s. And I was at IBM at the time.
As a result I decided to leave software development as a profession in 1989. It was clear to me by the end of 1989 that software development would never be a respected profession. The position I resigned was as the youngest ever system architect in IBM Australia at just the age of 25. I was SA for a 400 work month USD4M project to be implemented into 26 countries.....at 25.
In 1990 I transitioned to IBM Marketing. My job was to improve the productivity of IT shops using IBM hardware and software in the Insurance sector in Sydney Australia. It took a full year to go through the transition before I was allowed to speak to a customer. And I was ALREADY 8 years in IT and ALREADY the youngest system architect in the country. It didn't matter. I had to go through Marketing School and SE School like all the graduates. There was no short cut for me.
As it happened one of my first IBM customers wanted to implement Ralph Kimballs Data Interpretation System Software. I was assigned the job of implementing it in April 1991. The business results of using the data to support the management decision making process were so astonishing to us all that I changed focus from operational systems to what later became known as data warehousing.
A data warehouse is the single highest ROI investment a large company can make in the IT area. Indeed, in any area. Some of my customers doubled their profit using the ideas I gave them.
In 1991 I independently re-invented Ralphs idea of multi-level summaries in what would now be called "One Big Table". In 1992 I did a tour of all the big IBM DB2 customers who would accept my offer of a presentation of my new idea. I also spoke at all the DB2 user conferences on behalf of IBM presenting this idea.
Over the next few years my idea became widely adopted in Australia. I did some meetings with the National Australia Bank in 1992 and taught them about this idea which they then adopted. In the 1992-1997 period the NAB went from 1 billion dollars profit per year to 2 billion dollars profit per year and that was only possible with me giving them my ideas.
In 1993 I went to the Metaphor Users Conference in San Francisco and learned about the whole idea of dimensional models. Companies like Coca Cola and Proctor and Gamble were running Ralphs multi-level dimensional models.
In 1994-5 my customer did a pure research project to learn how to build dimensional models. That is where I learned to build dimensional models. That customer never went ahead with the implementation due to politics.
By 1996 I was launching the Hitachi Data Systems Asia Pacific data warehousing practice. We won the biggest data warehouse deal in the country in 1997, that being the Australian Customs Service. I was the man who introduced dimensional modeling to Australia and made it widely popular.
By 1996 it was obvious to me that data warehousing was the single biggest ROI investment a company could make and I was determined to be at the leading edge of that industry.
Bill, Ralph, Sean Kelly and I did our very best to promote honesty and integrity and not have the same happen to data warehousing as happened to software development.
We lost that argument too.
By 2002 it was clear that the area of data warehousing was going to go the same way as the area of software development. Even Bill has recently written that data warehousing has a bad reputation because so many men sold and delivered what they called a "data warehouse" that wasn't.
The IT industry overall has a very poor reputation. Indeed the Royal Bank Of Scotland had an outage where customers could not use their cards for payments for THREE DAYS. There was the recent event where literally planes were grounded because of a windows update failure.
The quality of software and the quality of data is terrible in large enterprises.
And now with the advent of AI this is being exposed. All over the internet there are people posting about how the data in companies is "not ready for AI" meaning that the data warehouses are so bad that if you put AIs on the top of them they are very likely to say things that are not true because of the very poor quality of the data.
One other thing I see very often is this.
And you can just go ask Ben at "Seattle Data Guy". Men who are self described "data engineers" and "data analysts" constantly publicly complain that they spend weeks putting together what they call "business cases" that are not only for the business case not to be accepted.
They all ask: "what can we do to have our business cases more commonly accepted?"
Only problem is they refuse to take advice from me and I was THE most successful data warehouse salesmen in Australia for quite a few years.
MEN on the IT side of the house want to have the people, both men and women, on the business side of the house buy their proposals without even being honest, let alone good at their jobs.
Their failure to be able to persuade the business side of the house to invest in their proposals, so called "business cases", is exactly because they are so dishonest they try and use unearned titles to gain credibility rather than hard work and experience.
The whole "IT" segment is in very poor shape with software crashes and bad data so common it is accepted as normal. And even now the IT segment is hyping up AI like it hyped up Hadoop in 2010 like it hyped up the internet in 1999.
And even today, the men in the IT segment will not take the advice of "honesty is the best policy".
Indeed, in case you haven't noticed, the idea that it's better to lie to people to gain money and influence had become very obvious in areas such as politics and big business. The idea of "honesty is the best policy" was long ago abandoned in favour of "if only you can fake honesty and sincerity you can make the sale".
The western world reflects the very low level of honesty in all areas of society now. Social trust is collapsing and social cohesion is collapsing and that bodes very poorly for the future of the west.
All because honesty and integrity were abandoned long ago.
Hi Andreas, it is not an “attack” to speak the truth. If men who call themselves “data engineers” wish to be respected by the men who are on the business side of they house they would have taken my advice.
That you falsley call good advice an “attack” is exactly why men like you are not respected on the business side of the house.
I have been in IT 44 years now. It is right and proper that any man who uses the title “engineer” who has not earned it be criticised for doing so.
I was an IBM Systems Engineer in the early 90s. I can assure you that it was VERY HARD to get through the IBM school to be called an associate SYSTEMS ENGINEER and VERY HARD to earn my promotion to the full level.
If men wish to bestow on themselves the title of “Data Enigineer” then they should be willing to defend themselves from my criticism and prove they earned the title. You just did the exact opposite. You played the “victim of an attack” like women do. You didn't defend the use of the title you use by demonstrating how you earned it.
Remember, in Roman times bridge builders stood beneath the bridge when it was stress tested to the full weight the bridge builder said it could carry. A real engineer often stakes his life and his reputation on the quality of his work. Even more often, he stakes other peoples lives on the quality of his work. Self described “data engineers” do not even allow peer review of their work because they know it is so bad.
As I said. If self described “data engineers” wished to be professionally respected by the business side of the house they would take my advice. I certainly have zero respect for men who call themselves data engineers and I wrote a relational database manager in 1985 as my university project.
I have been a global thought leader in data warehousing since the late 90s. And yes, I was also a system architect for operational systems in the 80s.