hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?

Cristina Lapa

hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?

hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data? if so, what do data modelers do then? I have been asked to write about roles and responsibilities for data modelers in Hadoop but everything I read in the internet, there is no data modeler involvement... maybe in the integrated layer?

thanks
Cristina

Eric Schiller

RE: hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?
(in response to Cristina Lapa)

From what I can tell based on research/some exposure, modeling still happens but tends to be more abstracted than usual.  Generally the three touchpoints I see are:

-- Canonical modeling of what the data represents and how it works across platforms to keep it relatively consistent.  

-- Modeling for NoSQL databases in the mix that is direct to schema development, and remodeling as the usage of the data derives what the most efficient physical model is as the common usage becomes clear.

-- Traditional modeling downstream of the main pipelines as the cleaner data still needs to be pulled into a data warehouse or similar analytics platform.

 

The biggest takeaway I got from EDW is that modeling is still important, and needs to adjust to gain understanding of the entire architecture, but also get used to being agile and knowing that the model will change frequently over time.

Cristina Lapa

[External] hi everyone. I am looking for the answer for this: do data modelers get involved at all i
(in response to Eric Schiller)
thank you so much Eric!!!








From:        Eric Schiller <[login to unmask email]>
To:        [login to unmask email]
Date:        03/29/2019 09:18 AM
Subject:        [External][Big Data] - RE: hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?



From what I can tell based on research/some exposure, modeling still happens but tends to be more abstracted than usual.  Generally the three touchpoints I see are:
-- Canonical modeling of what the data represents and how it works across platforms to keep it relatively consistent.  
-- Modeling for NoSQL databases in the mix that is direct to schema development, and remodeling as the usage of the data derives what the most efficient physical model is as the common usage becomes clear.
-- Traditional modeling downstream of the main pipelines as the cleaner data still needs to be pulled into a data warehouse or similar analytics platform.

The biggest takeaway I got from EDW is that modeling is still important, and needs to adjust to gain understanding of the entire architecture, but also get used to being agile and knowing that the model will change frequently over time.


Site Links: View post online   View mailing list online   Start new thread via email   Unsubscribe from this mailing list   Manage your subscription  

This email has been sent to: [login to unmask email]



Attachments

  • blankfilename.txt (12.7k)

Pat O'Sullivan

RE: [External] hi everyone. I am looking for the answer for this: do data modelers get involved at all i
(in response to Cristina Lapa)

Hi,
I agree with the comments from Eric - that is definitely similar to the different areas of modelling that we are seeing as well.  One additional point/question on the "data modeling for NoSQL" topic  : 

One big question we keep seeing is "under what circumstances and what business use cases does it make sense to deploy models in a schema-on-write fashion onto NoSQL/Hadoop".  In other words while we know that we can technically deploy a physical schema onto NoSQL,  is that something that actually makes sense ?  Especially when you consider the associated cost of building (and maintaining) the ETL to populate the new schema.  The concerns/questions that we see popping up include :

  • If you are going to the trouble and cost with such Schema-on-Write structures, then why not do so with a target such as a high-performant column store ?
  • Is not such a schema-on-write approach counter to the schema-on-read ethos of many NoSQL/Hadoop deployments?  

It feels like this is one area where the dust is still settling a little on what is specific best practice and there seems to be a lot of different opinions from different practitioners out there. I would be interested in hearing other views that folks might have about the dos and don'ts of modeling Schema on Write on NoSQL/Hadoop. 

William McKnight

RE: hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?
(in response to Cristina Lapa)

There is the work of understanding the different profiles of data that could be loaded. This is not traditional modeling, but your business teams will be delayed in planning how to use the data if they don't have an idea of the data that will be made available. Furthermore, there is the work of determining the data load into potentially multiple clusters. You could load all data into one cluster, but that's not always the best design.

Cristina Lapa

[External] hi everyone. I am looking for the answer for this: do data modelers get involved at all i
(in response to William McKnight)
thank you






From:        William McKnight <[login to unmask email]>
To:        [login to unmask email]
Date:        04/18/2019 10:05 AM
Subject:        [External][Big Data] - RE: hi everyone. I am looking for the answer for this: do data modelers get involved at all in big data?



There is the work of understanding the different profiles of data that could be loaded. This is not traditional modeling, but your business teams will be delayed in planning how to use the data if they don't have an idea of the data that will be made available. Furthermore, there is the work of determining the data load into potentially multiple clusters. You could load all data into one cluster, but that's not always the best design.


Site Links: View post online   View mailing list online   Start new thread via email   Unsubscribe from this mailing list   Manage your subscription  

This email has been sent to: [login to unmask email]



Siya Carla, Solution Consultant

Do data modelers get involved at all in big data?
(in response to Cristina Lapa)


Data modeling plays an important role in big data analytics. In big data, the process of storing data is called data modeling. Data modelers help translate the requirements of the businesses into logical, conceptual, and physical data models. It further help reduce data redundancy and improves the way in which data moves from a system to another.

Android App Development Company | Java Application Development Company