Recipe to migrate Drupal 7 Organic Groups to Drupal 9 Group

Despite there are some guides about migrating from OG to Group, as the well written https://thinktandem.io/blog/2018/03/30/migrating-drupal-7-organic-groups-to-drupal-8-group/ , and other guidelines such as https://www.drupal.org/project/group/issues/2826544 companion of https://gitlab.com/geeks4change/hubs4change/h4c/-/tree/608842e73cc02c45deede42556693a9db2a7fc37/modules_dev/h4c_migrate/config/install or https://www.drupal.org/node/2797845 . It's really adapted to your environment how you gonna carry on.

This is a short and crash guide that we at communia have done to complete the migrations. When Carrying on this migration must be estimated if the effort of writing code, in a form of migrate plugin classes or yml's, is worth enough than cooking csv's using spreadsheet skills. We've done both.

After importing everything from a standard drupal migration from 7 to 8(with what core offers), the groups types must be created, also their group roles and their group content types in a manual way. The site to be recreated is a news portal that takes headlines from several sites, and each blog or external blog is administrated in group, so being a group can be useful to add dynamic members or authors. So in the example I need to create two group types ( for each node enabled as organic group in d7), bundles are (machine)named:

  • blog(it was wrongly named previously bloc). I want also to add a role for each group type as admin to add granularity in permissions. So is needed to create the role admin for each type named:
  • blog_group-admin for blog group types.

I also marked the automatic role assignment for creators as admin role. It carries on a little of complexity in membership migrations as we must take care that memberships aren't duplicated (hence the plugin created below), as one membership will be created when groups are created on groups migration step, and another one when we import the memberships on memberships migration step.

So we need a simple module to add the migration configurations in yml, and it will aid us to create some extra plugins and hang on some hooks as hook_migrate_prepare_row. The module tree:

.
├── group-content-export.csv.template
├── group-memberships.csv.template
├── migrations
│   ├── d7_node_complete_og_to_canal_blog_extern.yml
│   ├── d7_node_complete_og_to_canal_proposta_externa.yml
│   ├── rename_d7_node_complete_blog_post.yml
│   ├── rename_d7_node_complete_podcast.yml
│   ├── upgrade_d7_node_og_blog_content.yml
│   ├── upgrade_d7_node_og_blog_translation.yml
│   ├── upgrade_d7_node_og_blog.yml
│   └── upgrade_d7_user_og_blog_memberships.yml
├── og_migrate_group.info.yml
├── og_migrate_group.module
├── README.md
└── src
    └── Plugin
        └── migrate
            ├── process
                ├── GroupCreator.php
                ├── SkipCreator.php
                └── SkipOnNotEmpty.php

Go to gitlab.com to get the actual code for each file: https://gitlab.com/communia/og_migrate_group

So now let's yamling.

Groups

As Organic groups usually was assigned to nodes (if that's not the case cook it as you need), you 'll see it will be based in d7_node migration, so d7_node must be migrated before.

migrations/upgrade_d7_node_og_blog.yml:

langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_og_blog
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
migration_group: insert_group
label: 'Insert OG Bloc group'
source:
  plugin: d7_node
  # The node bundle that must be get from source
  node_type: bloc
process:
  type:
    plugin: default_value
    # put blog as group type (it can be simple mapped here but this is to show how default values work)
    default_value: blog
  id: nid
  # The next uid will be the creator, if default role is marked it will be assigned to this user in group realm
  uid: node_uid
  label: title
  path: alias
  field_description: body
  # An illustration of how an image in a group could be used, images must be migrated before as usual (in this example it is not using media module).
  field_imatge_blogextern:
    plugin: sub_process
    source: field_imatge_blogextern
    process:
      target_id: fid
      alt: alt
      title: title
      width: width
      height: height
  # Here an illustration of how a term could be assigned, once again terms must exist before migrating this.
  field_tipus_bloc:
    plugin: sub_process
    source: taxonomy_vocabulary_7
    process:
      target_id:
        plugin: entity_lookup
        source: tid
        value_key: tid
        bundle_key: vid
        bundle: vocabulary_7
        entity_type: taxonomy_term
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
destination:
  # It's a group
  plugin: 'entity:group'
migration_dependencies: null

Once module is installed this migration can be imported with:

vendor/drush/drush/drush mim upgrade_d7_node_og_blog

and rolled back with:

vendor/drush/drush/drush migrate:rollback upgrade_d7_node_og_blog

if error locks any migration state saying it's currently importing (when is not) it can be fixed with:

drush migrate:reset-status upgrade_d7_node_og_blog

It's common that sometimes a crufty translation exists, in this example we found two extra fields that covered the translation, instead of using entity_translate or node_translate so field_titolbloces means title field in spanish and field_descripciobloces means field_description in spanish . This configuration will update existing groups translating it. Must be noted that group type must be marked as translatable editing the group type settings. migrations/upgrade_d7_node_og_blog_translation.yml:

langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_og_blog_translation
class: Drupal\migrate\Plugin\Migration
migration_tags:
  - 'Drupal 7'
migration_group: insert_group
label: 'Insert OG blog group translation'
migration_dependencies:
  required:
    - upgrade_d7_node_og_blog
source:
  plugin: d7_node
  node_type: 'bloc'
process:
  # Will update the previously imported groups
  id:
    plugin: migration_lookup
    source: nid
    migration: upgrade_d7_node_og_blog
  # Where can be found the label in spanish
  label:
    plugin: get
    source: field_titolbloces
  # Where can be found the description in spanish
  field_description:
    plugin: get
    source: field_descripciobloces
  # default value of langcode
  langcode:
    plugin: default_value
    default_value: es
  type:
    plugin: default_value
    default_value: blog
destination:
  plugin: 'entity:group'
  translations: true

This migration can be imported with:

vendor/drush/drush/drush mim upgrade_d7_node_og_blog_translation

and rolled back with:

vendor/drush/drush/drush migrate:rollback upgrade_d7_node_og_blog_translation

if error locks any migration state saying it's currently importing (when is not) it can be fixed with:

drush migrate:reset-status upgrade_d7_node_og_blog_translation

Memberships

As Organic groups memberships usually was assigned to users (if that's not the case cook it as you need), you 'll see it will be based in d7_user migration, so d7_user must be migrated before.

I found that here is easier to use csv, than to carry on too much code to take from here and there, so I have created a view of og memberships with data export in source site. View fields are:

  • OG membership: Og membership ID (id)
  • OG membership: Entity id (uid)
  • OG membership: Group ID (gid)
  • (Content entity referenced from og_user_node) Content: Identificator (UID) of the author (gid_uid)

View relationships are:

  • OG user node target id. (Content entity referenced from og_user_node) : A bridge to the Content entity that is referenced via og_user_node
  • OG membership: User from OG membership (REQUIRED)

If you prefer the sql of that view was:

SELECT og_membership.id AS id, og_membership.etid AS og_membership_etid, og_membership.gid AS og_membership_gid, node_og_membership.uid AS node_og_membership_uid, 'membership_export:page' AS view_name
FROM
{og_membership} og_membership
LEFT JOIN {node} node_og_membership ON og_membership.gid = node_og_membership.nid
LEFT JOIN {field_data_og_user_node} node_og_membership__field_data_og_user_node ON node_og_membership.nid = node_og_membership__field_data_og_user_node.og_user_node_target_id AND (node_og_membership__field_data_og_user_node.entity_type = 'user' AND node_og_membership__field_data_og_user_node.deleted = '0')
LEFT JOIN {users} og_user_node_node ON node_og_membership__field_data_og_user_node.entity_id = og_user_node_node.uid
LEFT JOIN {field_data_og_group_ref} node_og_membership__field_data_og_group_ref ON node_og_membership.nid = node_og_membership__field_data_og_group_ref.og_group_ref_target_id AND (node_og_membership__field_data_og_group_ref.entity_type = 'user' AND node_og_membership__field_data_og_group_ref.deleted = '0')
LEFT JOIN {users} og_group_ref_node ON node_og_membership__field_data_og_group_ref.entity_id = og_group_ref_node.uid
INNER JOIN {users} users_og_membership ON og_membership.etid = users_og_membership.uid AND (og_membership.entity_type = 'user')
LEFT JOIN {profile} profile_users ON users_og_membership.uid = profile_users.uid

Anyway it must output what's in group-memberships.csv.template. Here the csv fields

  • id The id of the membership(rarely used, but preserved)
  • uid, The user id of the membership (so the member).
  • gid, The id of the group that user is member.
  • gid_uid, The creator of the group (not used in migration as we take it with a dedicated migration plugin).
"id","uid","gid","gid_uid"
"46","1","14829","1"
"50","4","14814","4"
"52","12","14834","1"

And here the migration that carries on:

langcode: en
status: true
dependencies: {  }
id: upgrade_d7_user_og_blog_memberships
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7 og'
migration_group: insert_group
label: 'Insert Group Membership'
source:
  plugin: csv
  # Must be obtained from source with some exporter (e.g. views data export) and placed in this path
  path: 'public://imports/group-export.csv'
  header_row_count: 1
  ids:
    -
      id
  column_names:
    -
      id: Identifier
    -
      uid: UserID
    -
      gid: Group
  delimiter: ','
  enclosure: '"'
process:
  label: uid
  group_creator:
  # TO illustrate a plugin (not useful in the process),  that transforms from a group id to the uid of the group author.
    -
      plugin: group_creator
      source: gid
#    -
#      plugin: callback
#      callable: var_dump
  # Now will check that group is there via previous migration lookup (the first one we did):
  gid:
    -
      plugin: migration_lookup
      migration: upgrade_d7_node_og_blog
      no_stub: true
      source: gid
    -
      plugin: skip_on_empty
      method: row
      message: 'gid is missing'
  entity_id:
    # To illustrate how the previous `group_creator` could be injected here( not useful):
    #-
    #  plugin: callback
    #  callable: var_dump
    #  source: "@group_creator"
    # The plugin created to stop creating membership if current row user uid is the same as the group creator, to prevent duplicated members:
    -
      plugin: skip_creator
      user_key: uid
      gid_key: gid
      method: row
    # Finally check if user uid exists (probably if we are in this process point):
    -
      plugin: migration_lookup
      migration: upgrade_d7_user
      no_stub: true
      source: uid
    -
      plugin: skip_on_empty
      method: row
      message: 'uid is missing'
  # We hard code the role for all the migrated members:
  group_roles:
    plugin: default_value
    default_value: blog-group_admin
  type:
    plugin: default_value
    default_value: blog-group_membership
destination:
  # Memberships are considerated group_content also.
  plugin: 'entity:group_content'
migration_dependencies:
  optional:
    - upgrade_d7_node_og_blog
    - d7_user

Note the use of skip_creator plugin to stop creating membership if current row user uid is the same as the group creator, to prevent duplicated members. The code of src/Plugin/migrate/process/SkipCreator.php is:

<?php

namespace Drupal\og_migrate_group\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\MigrateException;
use Drupal\migrate\MigrateSkipRowException;
use Drupal\migrate\MigrateSkipProcessException;
use Drupal\migrate\Row;

/**
 * Skip if user_key as uid is the creator of group with gid as in gid_key.
 *
 * @MigrateProcessPlugin(
 *   id = "skip_creator"
 * )
 *
 * To skip if creator use the following:
 *
 * @code
 * field_creator:
 *   plugin: skip_creator
 *   user_key: uid
 *   gid_key: gid
 *   method: row
 * @endcode
 *
 */
class SkipCreator extends ProcessPluginBase {

  /**
   * Stops processing the current property when user_key as uid is the creator of group with gid as in gid_key.
   *
   * @param mixed $value
   *   The input value.
   * @param \Drupal\migrate\MigrateExecutableInterface $migrate_executable
   *   The migration in which this process is being executed.
   * @param \Drupal\migrate\Row $row
   *   The row from the source to process.
   * @param string $destination_property
   *   The destination property currently worked on. This is only used together
   *   with the $row above.
   *
   * @return mixed
   *   The input value, $value, if it is not empty.
   *
   * @throws \Drupal\migrate\MigrateSkipProcessException
   *   Thrown if the source property is not set and rest of the process should
   *   be skipped.
   */
  public function process($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $group = \Drupal::entityTypeManager()
      ->getStorage('group')
      ->load($row->getSource()[$this->configuration["gid_key"]]);
    if (!$group) {
      throw new MigrateException('gid not found.');
    }
    if ($group->getOwner()->id() == $row->getSource()[$this->configuration["user_key"]]) {
      throw new MigrateSkipRowException("Creator is the same uid, so it exists");
    }
    return $value;
  }

  /**
   * Skips the current row when user_key as uid is the creator of group with gid as in gid_key.
   *
   * @param mixed $value
   *   The input value.
   * @param \Drupal\migrate\MigrateExecutableInterface $migrate_executable
   *   The migration in which this process is being executed.
   * @param \Drupal\migrate\Row $row
   *   The row from the source to process.
   * @param string $destination_property
   *   The destination property currently worked on. This is only used together
   *   with the $row above.
   *
   * @return mixed
   *   The input value, $value, if it is not empty.
   *
   * @throws \Drupal\migrate\MigrateSkipRowException
   *   Thrown if the source property is not set and the row should be skipped,
   *   records with STATUS_IGNORED status in the map.
   */
  public function row($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $group = \Drupal::entityTypeManager()
      ->getStorage('group')
      ->load($row->getSource()[$this->configuration["gid_key"]]);
    if (!$group) {
      throw new MigrateException('gid not found.');
    }
    if ($group->getOwner()->id() == $row->getSource()[$this->configuration["user_key"]]) {
      throw new MigrateSkipRowException("Creator is the same uid, so it exists");
    }
    return $value;
  }

}

So now we have groups and their members. Now we can start thinking about the content.

Content

Here instead of doing everything at once (create nodes and add as content to groups) as in https://thinktandem.io/blog/2018/03/30/migrating-drupal-7-organic-groups-to-drupal-8-group/ , it will be based in a previously migrated node set, as it was did in previous step where from users it created group memberships.

Let's create the group content first setting that content as available in group type settings/content, in this example blog group available content will be proposta, blog_post(as blog entry), podcast_episode(as audio entry).

The next step starting from commit 9b9c731e it's easier than csv import, so it can be used with the plugin d7_og_content .

Via direct sql query

for group content imports (relation maker):

drush mim rename_d7_node_complete_blog,rename_d7_node_complete_podcast drush mim d7_og_blog_content

It will do the same as it's done in next csv import but using sql to previous import as source

Via view csv import

Despite we could create a new plugin which transforms from nid to title or uid, may be useful  to do it in source view group_content csv, so including these properties. Relevant view used to obtain source data will be:

Relations:

  • OG membership: Content from OG membership: 'og_membership_related_node'; REQUIRED
  • Entity Reference : Referenced Entity: 'og_group_ref_target_id';

Fields:

  • id, label: 'og_membership_id'
  • (relationship: og_membership_related_node_1)->nid , label:content_nid
  • (relationship: og_membership_related_node_1)->type, label: content_bundle
  • (relationship: og_membership_related_node_1)->uid, label: content_uid
  • (relationship: og_membership_related_node_1)->title, label: content_title
  • gid, label: 'gid'
  • (relationship:og_group_ref_target_id)->type, label: group_type

(With the filters and sorts adapted to your case, to prevent listing non-migratable content)

This will output

"og_membership_id","content_nid","content_bundle","content_uid","content_title","gid","group_type"
"13397","59546","blog","438","Good news","22477","bloc"
"13426","77","podcast","6","El sistema universitari","22447","bloc"
"13427","86","proposta","16","Estrobaca","22455","bloc"
"13428","87","blog","8","ssl","22449","bloc"

Here the yml that will process this csv:

langcode: en
status: true
dependencies: {  }
id: upgrade_d7_node_og_blog_content
class: null
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - CSV
migration_group: insert_group
label: 'blog Group Content Migration from CSV'
source:
  plugin: csv
  # Must be obtained from source with some exporter (e.g. views data export) and placed in this path
  path: 'public://imports/group-content-export.csv'
  header_row_count: 1
  ids:
    -
      og_membership_id
  column_names:
    -
      og_membership_id: Identifier
    -
      content_nid: NodeID
    -
      content_bundle: bundle
    -
      content_uid: uid
    -
      content_title: title
    -
      gid: group
    -
      group_type: GroupType
  delimiter: ','
  enclosure: '"'
process:
  label:
    # TO show the current item to debug
    #-
    #  plugin: callback
    #  callable: var_dump
    #  source: content_title
    -
      plugin: get
      source: content_title
  type:
    plugin: static_map
    source: content_bundle
    bypass: false
    # Customize this to match your values and group content plugins
    map:
      proposta: blog-group_node-proposta
      blog: blog-group_node-blog_post
      podcast: blog-group_node-podcast_episode
    default_value: blog-group_node-blog_post
  gid: gid
  entity_id:
    # To skip if element is not found (may be merged during translation
    # nodes migration fixes)
    -
      plugin: get
      source: content_nid
    -
      plugin: entity_lookup
      entity_type: node
      value_key: nid
    -
      plugin: skip_on_empty
      method: row
  uid: content_uid
destination:
  plugin: 'entity:group_content'
migration_dependencies: null

And that's all, after that Groups, Memberships and Content will be migrated.

Bonus track - rename machine name of a bundle when migration has been already executed.

First of all we create the bundle giving the desired name, and with the module field_tools we can make the clone of view_modes first and after that the fields clones. Once it is identical, we must rollback the migration of the content a given bundle: drush migrate:rollback d7_node_complete:podcast and then copy from web/sites/default/files/config__secret_directory_that _must_be_uniquely_and_obscured_from_public_found_at_settings/sync/migrate_plus.migration.upgrade_d7_node_complete_podcast.yml to a dedicated custom module in its migrations directory, then modify file setting the destination to desired bundle machine_name, also must be changed the id and filename according to what you are doing(in this case we set to rename_d7_node_complete_podcast), then run drush migrate:rollback d7_node_complete:podcast --verbose && drush mim rename_d7_node_complete_podcast --verbose to import again the content that we want. Any configuration over it must be executed once again, for example url_redirects migrations.

 

 

Bonus track 2 - node feeds in D7 to feeds_feed entity in D8

First create the feed types and add the fields as it was defined in source nodes that were acting as importers, then edit add the migration configuration, two working samples are in migrations:

  • d7_node_complete_og_to_canal_blog_extern.yml
  • d7_node_complete_og_to_canal_proposta_externa.yml

Both are based on the stock migration configurations that migrateplus has created and exported in config sync directory: upgrade_d7_node_complete_[bundle_of_source] . You can edit to create the feed entity instead of a node. Be careful to rollback previous imported bundle nodes migrations that match (in example it was feed and proposta_feed bundles). Flush plugins cache, and after that drush mim d7_node_complete_node_to_[bundle_of_source] will do the magic. This is the first step, Now you have two options because  feed source url is not easily picked.

One way and recommended as it can be automatizated, is using a new source plugin to fix the feed urls, look at commit  this is a new plugin that will lookup the feed in source d7 feed table and "magically" will provide the url using some config as this:

 

langcode: en
status: true
dependencies: {  }
id: d7_feed_sources
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7 feeds'
  - 'resync'
migration_group: resync
label: 'Drupal 7 Feeds sources'
source:
  plugin: d7_feed
process:
  fid:
    -
      plugin: entity_lookup
      source: title
      value_key: title
      entity_type: feeds_feed
#    -
#      plugin: callback
#      callable: var_dump
  source: source
destination:
  plugin: 'entity:feeds_feed'
migration_dependencies:
  optional:
    - d7_node_complete_og_to_canal_blog_extern
    - d7_node_complete_og_to_canal_proposta_externa

 

The other way is using csv in a manual way,  first instead of struggling too much I end up creating a view in source that outputs to csv these fields:

  • Content: title (title)
  • Feeds source: Source (source)
  • Content: type (bundle)
  • Content: Nid (nid)

Which csv I copied to public://imports/export-feeds.csv And created the auxiliary migration configuration to import from csv:

langcode: en
status: true
dependencies: {  }
id: fix_feeds_sources
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7 feeds'
migration_group: fix_feeds
label: 'Fix Feeds sources'
source:
  plugin: csv
  path: 'public://imports/export-feeds.csv'
  header_row_count: 1
  ids:
    -
      nid
  column_names:
    -
      title: title
    -
      source: Source
    -
      bundle: bundle
    -
      nid: Identifier
  delimiter: ','
  enclosure: '"'
process:
  fid:
    -
      plugin: entity_lookup
      source: title
      value_key: title
      entity_type: feeds_feed
#    -
#      plugin: callback
#      callable: var_dump
  source: source
destination:
  plugin: 'entity:feeds_feed'
migration_dependencies:
  optional:
    - d7_node_complete_og_to_canal_blog_extern
    - d7_node_complete_og_to_canal_proposta_externa

then drush mim fix_feeds_sources again. Then create the feed mappings as it was done in source(or modify as you want). And start importing

Add new comment

The content of this field is kept private and will not be shown publicly.

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.