How do I break up my data models into smaller chunks?

Michelle Knight

How do I break up my data models into smaller chunks?

How do I break up my data models into smaller chunks? This question came up at the Data Modeling Best Practices - Business and Technical approaches webinar today.

From what I heard, Donna Burbank cautioned about getting into too much detail with Data Modeling. She advised against having Death by Data Modeling (The three walls with complicated diagrams that take a couple of all-day meetings.) To be more effective, Donna suggested breaking these Data Models in chunks.

Responses to these questions mentioned breaking the business into subject areas, following the use case from the Agile Sprint, and focus on some core concepts. Does anyone else have thoughts or tips and tricks?

Presentation slides will be available by end of day Monday, 10/28/19.

Edited By:
Michelle Knight[All DATAVERSITY Members] @ Oct 24, 2019 - 03:42 PM (America/Pacific)
Michelle Knight[All DATAVERSITY Members] @ Oct 24, 2019 - 03:47 PM (America/Pacific)

Thomas Fontanella

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

I am afraid that I don't have solution ... but I completely agree with the problem! In our case we are trying to make data models available for our commercial software systems. But no one needs to see the whole model - and we'd need to rent a billboard to display it anyways. The vendors have not been able to provide those subject area chunks either. It seems like an obvious need.

Merrill Albert

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

I don't think huge wall-sized diagrams benefit anyone.  It's too much to absorb, too convoluted to follow lines, and too time-consuming to manage.  Subject areas are so much easier.  Each subject area is a topic of conversation.  You have not only the entities for that specific area, but all entities it touches in other subject areas.  That gives you the link to other diagrams to go to for understanding other topics.

Michelle Knight

RE: How do I break up my data models into smaller chunks?
(in response to Merrill Albert)

Thank you, Thomas and Merrill. Upon doing some research and writing the concept of knowledge graphs or property graphs seemed viable. I picked up some diagrams from Thomas Frisendal. From my software testing and information architecture experiences, I used Microsoft Visio to do this sort of mapping. I remember it clarified which concepts or operations could be chunked to make running a test or putting up a website more effective. Has anyone had hands-on experience with this sort of graphing? Would a property graph be a good tool to see data model per subject areas, Merrill?

See the attached image.

Michelle Knight

[login to unmask email]

Freelance Production Assistant

Freelance Data, Technology and Science Writer

Attachments

  • property_graph_example.png (111.1k)
  • property-graph-definition.png (150.6k)
Edited By:
Michelle Knight[All DATAVERSITY Members] @ Nov 26, 2019 - 09:57 AM (America/Pacific)

Merrill Albert

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

My preference is always logical relational models.

Walter Howard

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

For agile development, you build the data model to support the current sprint, no more, no less.  The size is based on the number of stories in the sprint.

Once the application is complete, the model may be quite large.

And Donna is correct, no one likes to sit through large complex data model reviews.

The solution I've adopted over the years is to take a process based review of the model.  This is akin to the same methodology that was used to build the data model.  Common processes such as Create new Customer, Take an Order, Create an Account, all make sense to the business community.  Grouping the underlying entities, attributes, and relationships by process should comprise a reasonable, comprehensible "chunk". 

Phil Forestall

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

Lots of good answers on this post. I suggest you start by breaking up the data modeling over time. Design from the general to the particular, i.e. first conceptual, then logical, then physical.

As much as conceptual data modeling can contribute to solution architecture, like all conceptual modeling it should also inform project planning and budgeting. A conceptual data model helps improve project planning early by :

  • defining data scope
  • defining common data vocabulary
  • sketching data complexity, volumes, sourcing and integration
  • revealing project demands for expertise and time allocation
  • generally helping to understand project issues and risks

Suggestions for conceptual data modeling:

  • buy, copy (legally!) or build a common conceptual data model for all development in your organization:
    • entity names to be business-oriented, definitions from respected outside sources; model includes credible references to external literature
    • entities-only (no attribution!) because when the focus turns to fields, nobody can talk about anything else, you're doing logical modeling, and the time requirement just blooms
  • divide the data space into subject areas:
    • subject areas are internally cohesive; they should be relatively self-contained
    • not too tightly bound to the other subject areas
    • maintain the subject areas in subsequent logical modeling
  • every development project links to this conceptual model as-is:
    • so the entities in play in the project are pre-defined
    • new entities from the project can be adopted, with thanks, in the conceptual model
    • changed entities are scrutinized very carefully; why is this project different? what are we learning here? this is important governance
    • so conceptual modeling takes days (if not hours) and cheap enough to do regardless of expected project size

There's no reason not to do a conceptual model for OTS product acquisition as well as new development.

Best wishes.

Thomas Fontanella

RE: How do I break up my data models into smaller chunks?
(in response to Phil Forestall)

Great advice from all - thank you! We have informal agreement on subject areas so it wouldn't be starting from a blank slate to associate entities to them. We are also in the process of selecting/deploying a data catalog w/business glossary which may help to document and communicate the model. 

Michelle Knight

RE: How do I break up my data models into smaller chunks?
(in response to Phil Forestall)

Thank you, Phil. It sounds that chunking data modeling is equivalent to describing requirements necessary to meet the overall business strategy. In a sense not only has the business data been modeled for technology or a particular project, but also as a rational to verify and validate business decisions. Also, data modeling can be used to describe specifications for agile sprints and to what developers and testers need to consider acceptable data behavior results.

Michelle Knight

[login to unmask email]

Freelance Production Assistant

Freelance Data, Technology and Science Writer

Edited By:
Michelle Knight[All DATAVERSITY Members] @ Dec 16, 2019 - 09:39 AM (America/Pacific)

William Frank

RE: How do I break up my data models into smaller chunks?
(in response to Walter Howard)

I agree that processes are more amenable to business people, and processes should match good sprints. and that a team should work on one set of processes at a time,(regardless of methodology).  This matches the bounded context for domain driven design, and works well for single responsibily microservices.   

But, the implication that the data model is *created* as part of a sprint flies in the face of the main reason data models exist in most institutions: to effect easy integration and partitioning of applications.   An overall conceptual data model informs to models for each application, so the models and the applications are done in a consistent manner.   There is more to a technology support system than applications delivered. 

I also agree that wall-sized models are pretty useless: a picture is worth a thousand words, unless the picture has a thousand symbols on it!   The data subjects can be broken into packages with dependencies from the high-level conceptual model.  These will be at cut points in what would be the overall model, so that the connections between data elements across packages are minimized.   

Easier to do in UML than in E/R.   The packages are first-class design elements.  The dependencies discovered also help immensely in having a rational development plan.  (For example, building transaction execution before parties and assets transacted is not recommended.  Packages can be drawn by hand or in a free form tool.  

Frank T van Amsterdam

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

In my experience, running a 'blueprinting' service at a global pharma organisation for the last 5 years, most business people don't have the time or the focus to analyse and discuss huge enterprise data models. Instead they are very happy with smaller (sub) models that focus on an epic, a use case, a business process, or a specific set of business requirements and provide a clear, understandable user story that aligns to their business problem. Building an enterprise model iteratively, ideally through a dedicated service team, helps linking multiple related use cases, and ensures all use cases adhere to the same data standards and entity/attribute definitions. We have seen that this helps improve data quality in larger domains, now more than ever necessary to enable machine learning and AI to leverage data at a larger scale.

William Frank

RE: How do I break up my data models into smaller chunks?
(in response to Frank T van Amsterdam)

I say ++, Frank.  

William McKnight

RE: How do I break up my data models into smaller chunks?
(in response to Michelle Knight)

In an agile world, I say don't model anything you aren't able to both populate and use in production in the next few months. That forces some smaller full life cycles into the planning when the tendency is always to bigger and stepwise.

Michelle Knight

RE: How do I break up my data models into smaller chunks?
(in response to William McKnight)

Thank you Merrill, Walter, Thomas, William, Frank, and William.

Michelle Knight

[login to unmask email]

Freelance Production Assistant

Freelance Data, Technology and Science Writer