squid by arc46

squid by arc46

Easily supply your peripheral systems with

the data they need

in the format they want.

Graphical Big Picture coming up...

We recognize that the long text below will not satisfy the "In 90 seconds: What does squid by arc46 actually do and how does it work?" question.

A visual and possibly animated big picture intro is currently being created by the same people that created our homepage.

ETA: Mid/End September.

In the meantime give the text a chance :-)

TL;DR

squid by arc46 enables the configurable export of all and any data including references, relations ( Owns::Owned By, etc.), "backreferences" (a subset of "related lists") and the various forms of M-to-N tables (cmdb_rel_teams, task_ci, etc.) in the ServiceNow database as a single structured JSON.

squid by arc46 comes with hundreds of predefined configurations making the app truly plug-and-play. And if these configurations don't match your needs they can be easily customized.

This enables the easy integration of any external data consumers, e.g. peripheral systems.

It does this in the most efficient way possible with the least impact on your ServiceNow production systems.

Can't ServiceNow do that out-of-the-box?

Sort of, kind of, but not really. ServiceNow gives you access to tables by way of the Table API. This returns a very rudimentary rendering of the requested data without the inclusion of any referenced data. Getting all the data you want and need will often require multiple requests and processing logic on the consuming side.

Comparison squid by arc46 <=> TableAPI

Let's compare how ServiceNow TableAPI and squid by arc46 solve the following data request

Give me all relevant data on the laptop the author of these lines is using. Please include

• asset information
• assigned_to including all information of the person the referenced in assigned_to
• location
• manufacturer
• model
• Application Service this ci is used by
• network ports of the laptop with name and netmask

squid by arc46 can do a lot more. This is just a first impression.

TL;DR - squid by arc46:1 - TableAPI:11

As you will see below squid by arc46 is able to satisfy the above data request in one JSON with one http request.

ServiceNow TableAPI on the other hand requires 11 request/response cycles and logic on the client side to achieve the same result.

Expand to see sequence of all actions required to satisfy the example data request with TableAPI

Retrieve initial cmdb_ci_pc_hardware JSON

• REST Request (1): Initial request to cmdb_ci_pc_hardware table.
  • Asset: Client Logic: Extract asset link from cmdb_ci_pc_hardware json.
  • Asset: REST Request (2): Retrieve link extracted above.
  • Assigned To: Client Logic: Extract assigned_to link from cmdb_ci_pc_hardware json.
  • Assigned To: REST Request (3): Retrieve link extracted above.
    • Assigned To - Location: Client Logic: Extract location link from sys_user json.
    • Assigned To - Location: REST Request (4): Retrieve link extracted above.
  • Location: Client Logic: Extract location link from cmdb_ci_pc_hardware json (This is the location of the ci, not the user).
  • Location: REST Request (5): Retrieve link extracted above.
  • Manufacturer: Client Logic: Extract manufacturer link from cmdb_ci_pc_hardware json.
  • Manufacturer: REST Request (6): Retrieve link extracted above.
  • Model: Client Logic: Extract model_id link from cmdb_ci_pc_hardware json.
  • Model: REST Request (7): Retrieve link extracted above.

Resolving Depends On::Used by relations

• Client Logic: Client must know that 1a9cb166f1571100a92eb60da2bce5c5 is the sys_id of Depends On::Used by cmdb_rel_type.
• REST Request (8): Resolve Depends On::Used by relation. Returns cmdb_rel_ci JSON.
  • Parent: Client Logic: Extract parent link from cmdb_rel_ci JSON.
  • Parent: REST Request (9): Retrieve link extracted above. The target is an unspecific cmdb_ci.
  • Parent: Client Logic: Extract correct sys_class_name from returned JSON.
  • Parent: Repeat REST Request (10) with correct target table cmdb_ci_service_auto.

Resolving network ports of the example laptop

• REST Request (11): Query cmdb_ci_network_adapter with cmdb_ci=940b0cff873379d06dc863540cbb355e.

squid by arc46

squid by arc46 is based on so-called configs that define how and with what content entities are rendered. These configs are highly flexible and allow for almost any form of JSON representation.

squid by arc46 is also able to include all properties of subclasses, e.g. a config cmdb_ci_server could include all fields defined on cmdb_ci_linux_server which contains the added property kernel_release, cmdb_ci_aix_server which contains the added property mainframe and cmdb_ci_solaris_server which contains the added properties ldom_role, ldom_state and ldom_version.

Requesting our example use case would look like this:

HTTP Request

https://yourinstance.servicenow.com/x_tdb_gen_read/v1/data/cmdb_ci_pc_hardware_full_inline?encodedQuery=base_name=w010y7&relations=used_by_service,network_ports_inline

Let's break down the components of the request:

cmdb_ci_pc_hardware_full_inline

is the requested configuration. By convention configurations should align with the ServiceNow class or tables that are requested. full_inline indicates that we want to retrieve all (most) data and that we want to display it inlined as opposed to referenced. (See TODO to better understand the difference.)

cmdb_ci_pc_hardware_full_inline references a ServiceNow database view that defines what properties are rendered. Any references may be rendered in one of multiple ways. For demonstrations purposes we are rendering all content inline.

An example of a fully resolved reference would be assigned_to. This would render as

{
  ...,
  "assigned_to": {
    "active": true,
    "email": "rainer.alfoeldi.2@systemx.ch",
    "first_name": "Rainer",
    "last_name": "Alföldi",
    "location": {
      "city": "Bern",
      "cmn_location_type": "room",
      "country": "Switzerland",
      "full_name": "CH/Bern/Strasse 12/Strasse 12 - C/5.OG/B / 504",
      "name": "B / 504",
      "street": "Strasse 12 - C",
      "sys_id": "483aad38872482903efcec230cbb357f",
      "sys_updated_on": "2024-02-21T16:38:28Z",
      "zip": "3000"
    },
    "sys_class_name": "sys_user",
    "sys_id": "c8b1c7a4874561103efcec230cbb354d",
    "user_name": "alfoeldirai"
  },
  ...
}

Please notice that the location referenced by sys_user is recursively resolved and rendered.

Further reference properties would be

• asset
• location - of the laptop as opposed to the user.
• manufacturer
• model_id

See the full JSON below for details.

encodedQuery

restricts the result set. encodedQuery may be any valid ServiceNow encodedQuery, with a few restrictions to prevent access controls from being circumvented. In this case we are requesting data for a specific laptop w010y

relations

tells squid by arc46 which of the pre-configured relations should be resolved. As this is potentially resource intensive (don't worry, squid by arc46 is highly efficient and will not crash your ServiceNow instance), we only resolved relations when explicitly requested.

Here we are requesting used_by_service and network_ports_inline. (squid by arc46 comes pre-configured for most of the standard use cases, but you can and probably will adapt and expand these and define your own.)

used_by_service will add any cmdb_ci_service that references the rendered ci by way of Depends On::Used by relation to a Used By JSON property.

{
  ...,
  "Used by": [
    {
      "name": "it.grp.mobile_workplace_performance",
      "service_classification": "Application Service",
      "sys_class_name": "cmdb_ci_service_auto",
      "sys_id": "cca1591d8778ce106dc863540cbb35a6"
    }
  ],
  ...
}

network_ports_inline will add any cmdb_ci_network_adapter that reference the rendered ci as cmdb_ci.

{
  ...,
  "network_ports": [
    {
      "name": "eth0",
      "netmask": "255.255.255.0",
      "sys_class_name": "cmdb_ci_network_adapter",
      "sys_id": "2b94d97b877fb1146dc863540cbb350e"
    },
    {
      "name": "wlan0",
      "netmask": "255.255.255.0",
      "sys_class_name": "cmdb_ci_network_adapter",
      "sys_id": "8744113f877379503efcec230cbb356b"
    }
  ],
  ...
}

Expand to see the full JSON returned by squid by arc46 for the above request.

{
  "metadata": {
    "Caveat": "THIS IS BETA CODE AND SUBJECT TO CHANGE AT ANY TIME!",
    "git": "DataRestApi-0.1.0+63-68df482-dirty",
    "build": "2024-03-10T13:00:39.981Z",
    "licensed_max_entities": "Unlimited",
    "config": "cmdb_ci_pc_hardware_full_inline",
    "request_received": "2024-03-10T13:14:33.463Z",
    "parse_config": "184ms",
    "query_start": "2024-03-10T13:14:33.747Z",
    "query_duration": "2ms",
    "query_returned": "2024-03-10T13:14:33.749Z",
    "row_count": 1,
    "limit": 1,
    "sysIds": [
      "940b0cff873379d06dc863540cbb355e"
    ],
    "combined_filter": "base_sys_idIN940b0cff873379d06dc863540cbb355e"
  },
  "data": [
    {
      "asset": {
        "display_name": "23282981 - Lenovo ThinkPad P16 Gen1",
        "sys_class_name": "alm_hardware",
        "sys_id": "d210687387b7f9d06dc863540cbb3506",
        "sys_updated_on": "2024-02-22T12:24:46Z"
      },
      "asset_tag": "23282981",
      "assigned_to": {
        "active": true,
        "email": "rainer.alfoeldi.2@systemx.ch",
        "first_name": "Rainer",
        "last_name": "Alföldi",
        "location": {
          "city": "Bern",
          "cmn_location_type": "room",
          "country": "Switzerland",
          "full_name": "CH/Bern/Strasse 12/Strasse 12 - C/5.OG/B / 504",
          "name": "B / 504",
          "street": "Strasse 12 - C",
          "sys_id": "483aad38872482903efcec230cbb357f",
          "sys_updated_on": "2024-02-21T16:38:28Z",
          "zip": "3000"
        },
        "sys_class_name": "sys_user",
        "sys_id": "c8b1c7a4874561103efcec230cbb354d",
        "user_name": "alfoeldirai"
      },
      "attestation_status": "Not Yet Reviewed",
      "attested": false,
      "can_print": false,
      "category": "Hardware",
      "cd_rom": false,
      "cost_cc": "USD",
      "fault_count": 0,
      "form_factor": "Laptop",
      "hardware_status": "installed",
      "hardware_substatus": "in_use",
      "install_status": 1,
      "internet_facing": true,
      "life_cycle_stage": "Operational",
      "life_cycle_stage_status": "In Use",
      "location": {
        "city": "Bern",
        "cmn_location_type": "room",
        "country": "Switzerland",
        "full_name": "CH/Bern/Strasse 12/Strasse 12 - A/SharedDesk/SharedDesk",
        "name": "SharedDesk",
        "street": "Strasse 12 - A",
        "sys_id": "870a2138872482903efcec230cbb3558",
        "sys_updated_on": "2024-02-21T16:37:51Z",
        "zip": "3000"
      },
      "manufacturer": {
        "city": "Morrisville",
        "country": "USA",
        "name": "Lenovo",
        "sys_class_name": "core_company",
        "sys_id": "aa0a6df8c611227601cd2ed45989e0ac"
      },
      "model_id": {
        "asset_tracking_strategy": "leave_to_category",
        "asset_tracking_unit": {
          "sys_id": "51a74add0f233300fee7579ac4767e59",
          "sys_class_name": "cmdb_model_unit"
        },
        "bundle": false,
        "certified": false,
        "cmdb_model_category": [
          "4fbbb127876f3d106dc863540cbb3519"
        ],
        "comments": "Mietmodell",
        "cost": "0",
        "customer_visible": false,
        "display_name": "Lenovo ThinkPad P16 Gen1",
        "energy_star": false,
        "life_cycle_stage": "To Be Determined",
        "life_cycle_stage_status": "To Be Determined",
        "manufacturer": {
          "city": "Morrisville",
          "country": "USA",
          "name": "Lenovo",
          "sys_class_name": "core_company",
          "sys_id": "aa0a6df8c611227601cd2ed45989e0ac"
        },
        "name": "ThinkPad P16 Gen1",
        "picture": "d4866ada87b04a503efcec230cbb35c4",
        "rack_units": 1,
        "salvage_value": "0",
        "status": "In Production",
        "sys_class_name": "cmdb_hardware_product_model",
        "sys_created_by": "import.systemx",
        "sys_created_on": "2023-02-14T14:17:19Z",
        "sys_id": "70a44cbf87f739d06dc863540cbb3524",
        "sys_mod_count": 72,
        "sys_updated_by": "import.systemx",
        "sys_updated_on": "2024-03-09T23:09:38Z",
        "type": "Generic",
        "u_virtual": false,
        "unit_of_measure_system": "us imperial"
      },
      "monitor": false,
      "name": "w010y7",
      "operational_status": 1,
      "serial_number": "PF3TE9JZ",
      "skip_sync": false,
      "subcategory": "Computer",
      "sys_class_name": "cmdb_ci_pc_hardware",
      "sys_created_by": "import.systemx",
      "sys_created_on": "2023-06-09T16:44:08Z",
      "sys_id": "940b0cff873379d06dc863540cbb355e",
      "sys_mod_count": 45,
      "sys_updated_by": "import.systemx",
      "sys_updated_on": "2024-03-09T23:27:48Z",
      "u_host_name": "w010y7",
      "u_third_party": false,
      "u_virtual": false,
      "unverified": false,
      "virtual": false,
      "Used by": [
        {
          "name": "it.grp.mobile_workplace_performance",
          "service_classification": "Application Service",
          "sys_class_name": "cmdb_ci_service_auto",
          "sys_id": "cca1591d8778ce106dc863540cbb35a6"
        }
      ],
      "network_ports": [
        {
          "name": "eth0",
          "netmask": "255.255.255.0",
          "sys_class_name": "cmdb_ci_network_adapter",
          "sys_id": "2b94d97b877fb1146dc863540cbb350e"
        },
        {
          "name": "wlan0",
          "netmask": "255.255.255.0",
          "sys_class_name": "cmdb_ci_network_adapter",
          "sys_id": "8744113f877379503efcec230cbb356b"
        }
      ]
    }
  ],
  "relations": {},
  "referenced": {}
}

ServiceNow TableAPI

The ServiceNow TableAPI will - eventually - give you all the data squid by arc46 gave you. It will just make you work harder to get it and some aspects (mainly security features) cannot be achieved by TableAPI at all.

HTTP Request

https://yourinstance.servicenow.com/api/now/table/cmdb_ci_pc_hardware?sysparm_query=name%3Dw010y7

cmdb_ci_pc_hardware

With TableAPI you are - nomen est omen - accessing a database table, not a configuration. As such you get - unless you restrict the returned fields by way of 'sysparam_fields' - all fields defined on the table.

sysparam_query=name%3Dw010y7

This is equivalent to the encodedQuery param provided by squid by arc46. (TODO check if TableAPI allows for dot-walking in the query.) If you look really closely you will notice that squid by arc46 defined the encodedQuery as encodedQuery=base_name=w010y7. This is due to the fact that squid by arc46 may aggregate multiple tables into a single view.

Rendering of referenced entities

TableAPI will render referenced entities as

{
  ...,
  "manufacturer": {
    "link": "https://yourinstance.servicenow.com/api/now/table/core_company/aa0a6df8c611227601cd2ed45989e0ac",
    "value": "aa0a6df8c611227601cd2ed45989e0ac"
  },
  ...
}

warning

This is of limited value as any details concerning the manufacturer have to be retrieved with a further request to the server. Resolving all references in our example would require multiple additional requests.

TableAPI will allow you to return the display_value of a referenced entity (sysparm_display_value=true):

{
  ...,
  "manufacturer": {
    "display_value": "Lenovo",
    "link": "https://yourinstance.servicenow.com/api/now/table/core_company/aa0a6df8c611227601cd2ed45989e0ac"
  },
  ...
}

This is of even more limited value as the display value is dependent on UI settings of the user (controllable), but also highly ambiguous.

This may be slightly mitigated by requesting sysparm_display_value=all which results in

{
  ...,
  "manufacturer": {
    "display_value": "Lenovo",
    "link": "https://yourinstance.servicenow.com/api/now/table/core_company/aa0a6df8c611227601cd2ed45989e0ac",
    "value": "aa0a6df8c611227601cd2ed45989e0ac"
  },
  ...
}

Not quite as ambiguous as only display_value, but not any more useful than the initial result and you pay for it with a highly verbose JSON:

{
  ...,
  "os_service_pack": {
    "display_value": "",
    "value": ""
  },
  "cpu_core_thread": {
    "display_value": "",
    "value": ""
  },
  "cpu_manufacturer": {
    "display_value": "",
    "value": ""
  },
  ...
}

Resolving relations and 'back-references'

In order to find out which services are associated with our example laptop, we have to explicitly query cmdb_rel_ci with

https://yourinstance.servicenow.com/api/now/table/cmdb_rel_ci?sysparm_query=child%3D940b0cff873379d06dc863540cbb355e%5Etype%3D1a9cb166f1571100a92eb60da2bce5c5

{
  "result": [
    {
      "connection_strength": "always",
      "parent": {
        "link": "https://yourinstance.servicenow.com/api/now/table/cmdb_ci/cca1591d8778ce106dc863540cbb35a6",
        "value": "cca1591d8778ce106dc863540cbb35a6"
      },
      "sys_mod_count": "7",
      "sys_updated_on": "2024-03-10 02:08:40",
      "sys_tags": "",
      "type": {
        "link": "https://yourinstance.servicenow.com/api/now/table/cmdb_rel_type/1a9cb166f1571100a92eb60da2bce5c5",
        "value": "1a9cb166f1571100a92eb60da2bce5c5"
      },
      "sys_id": "6cef41ed87b482506dc863540cbb35e1",
      "sys_updated_by": "import.systemx",
      "port": "",
      "sys_created_on": "2024-03-07 08:54:41",
      "x_t44t_tier44_em_8_cable": "",
      "percent_outage": "",
      "sys_created_by": "import.systemx",
      "child": {
        "link": "https://yourinstance.servicenow.com/api/now/table/cmdb_ci/940b0cff873379d06dc863540cbb355e",
        "value": "940b0cff873379d06dc863540cbb355e"
      }
    }
  ]
}

You then have to retrieve the link given at result.parent.link. As this resolves to cmdb_ci and not to cmdb_ci_service you would have to execute another request to retrieve any service specific information.

The same procedure would have to be executed in order to retrieve the network_ports returned by squid by arc46 in our example above.

Expand to see the full JSON returned by TableAPI for the above request.

{
  "result": [
    {
      "os_address_width": "",
      "attested_date": "",
      "operational_status": "1",
      "os_service_pack": "",
      "cpu_core_thread": "",
      "cpu_manufacturer": "",
      "sys_updated_on": "2024-03-09 23:27:48",
      "discovery_source": "",
      "first_discovered": "",
      "due_in": "",
      "gl_account": "",
      "invoice_number": "",
      "sys_created_by": "import.systemx",
      "ram": "",
      "warranty_expiration": "",
      "cpu_name": "",
      "cpu_speed": "",
      "owned_by": "",
      "checked_out": "",
      "disk_space": "",
      "sys_domain_path": "/",
      "business_unit": "",
      "object_id": "",
      "maintenance_schedule": "",
      "cost_center": "",
      "attested_by": "",
      "dns_domain": "",
      "assigned": "",
      "life_cycle_stage": {
        "link": "https://yourinstance.servicenow.com/api/now/table/life_cycle_stage?name=Operational",
        "value": "Operational"
      },
      "purchase_date": "",
      "cd_speed": "",
      "short_description": "",
      "floppy": "",
      "u_third_party": "false",
      "managed_by": "",
      "os_domain": "",
      "can_print": "false",
      "last_discovered": "",
      "sys_class_name": "cmdb_ci_pc_hardware",
      "cpu_count": "",
      "manufacturer": {
        "link": "https://yourinstance.servicenow.com/api/now/table/core_company/aa0a6df8c611227601cd2ed45989e0ac",
        "value": "aa0a6df8c611227601cd2ed45989e0ac"
      },
      "life_cycle_stage_status": {
        "link": "https://yourinstance.servicenow.com/api/now/table/life_cycle_stage_status?name=In+Use",
        "value": "In Use"
      },
      "vendor": "",
      "model_number": "",
      "assigned_to": {
        "link": "https://yourinstance.servicenow.com/api/now/table/sys_user/c8b1c7a4874561103efcec230cbb354d",
        "value": "c8b1c7a4874561103efcec230cbb354d"
      },
      "start_date": "",
      "os_version": "",
      "serial_number": "PF3TE9JZ",
      "cd_rom": "false",
      "support_group": "",
      "correlation_id": "",
      "unverified": "false",
      "attributes": "",
      "u_host_name": "w010y7",
      "asset": {
        "link": "https://yourinstance.servicenow.com/api/now/table/alm_asset/d210687387b7f9d06dc863540cbb3506",
        "value": "d210687387b7f9d06dc863540cbb3506"
      },
      "cpu_core_count": "",
      "form_factor": "Laptop",
      "skip_sync": "false",
      "x_t44t_tier44_em_8_aggregation": "",
      "most_frequent_user": "",
      "attestation_score": "",
      "sys_updated_by": "import.aixboms",
      "sys_created_on": "2023-06-09 16:44:08",
      "cpu_type": "",
      "sys_domain": {
        "link": "https://yourinstance.servicenow.com/api/now/table/sys_user_group/global",
        "value": "global"
      },
      "install_date": "",
      "asset_tag": "23282981",
      "hardware_substatus": "in_use",
      "fqdn": "",
      "change_control": "",
      "internet_facing": "true",
      "delivery_date": "",
      "hardware_status": "installed",
      "install_status": "1",
      "supported_by": "",
      "u_compliance_relevant": "",
      "name": "w010y7",
      "subcategory": "Computer",
      "default_gateway": "",
      "chassis_type": "",
      "virtual": "false",
      "assignment_group": "",
      "managed_by_group": "",
      "sys_id": "940b0cff873379d06dc863540cbb355e",
      "po_number": "",
      "checked_in": "",
      "sys_class_path": "/!!/!2/!(/!6",
      "mac_address": "",
      "company": "",
      "justification": "",
      "department": "",
      "u_label": "",
      "u_virtual": "false",
      "comments": "",
      "cost": "",
      "os": "",
      "attestation_status": "Not Yet Reviewed",
      "cmdb_ot_entity": "",
      "sys_mod_count": "45",
      "monitor": "false",
      "ip_address": "",
      "model_id": {
        "link": "https://yourinstance.servicenow.com/api/now/table/cmdb_model/70a44cbf87f739d06dc863540cbb3524",
        "value": "70a44cbf87f739d06dc863540cbb3524"
      },
      "duplicate_of": "",
      "sys_tags": "",
      "cost_cc": "USD",
      "order_date": "",
      "schedule": "",
      "environment": "",
      "due": "",
      "attested": "false",
      "location": {
        "link": "https://yourinstance.servicenow.com/api/now/table/cmn_location/870a2138872482903efcec230cbb3558",
        "value": "870a2138872482903efcec230cbb3558"
      },
      "category": "Hardware",
      "fault_count": "0",
      "lease_id": ""
    }
  ]
}

Do more with less

squid by arc46 does a whole lot more with a whole lot less.

This is only a small example of what squid by arc46 is capable of. See the full configuration documentation to learn more.

ServiceNow GraphQL (TODO)

TODO