Skip to main content

Data Butler

The following section describes all web based features, services and functionalities, that are part of Data Butler Transparency and Data Butler Quality contracts. Read more about the different packages on our Fleet Data website (https://hoppe-fleetdata.com).

Ship Management‚Äč

After successful login you will be presented with the fleet overview page:

image2022-8-26_7-55-18.png

The page displays a variety of information in a graphical representation. All individual green and red dots show ships of which we obtain data regulary.

In addition to the ship status indicators, the portal also provides a ship counter indicator at the top left corner of the map. This indicator shows a number at the center indicating total number of ships (e.g. 41 according to screen shot below) which are expected to transmit data to shore either via STS or via other means.

image2021-11-25_11-31-40.png

To find individual ships, just type their name or IMO number in the "Filter Ships by IMO Number or Name" field. Above this field, there is also the possibility to change views from the map visualization to a list by clicking in the "List" tab:

image2022-8-23_10-35-10.png

Here you can find an overview over the whole fleet without any effort, including information about connectivity status, IMO number, position, last connection and more:

image2022-8-23_10-46-22.png

Clicking on the ship will lead you to a pop-up drawer, that is described later on and provides a deep-dive into the ship's details.

Ship Status‚Äč

The status indicator of each ship has three components, a dot, an arrow indicating the heading/course of the ship as well as a small cloud symbol. The following table provides a key for how to interpret the different states of the ship. While the color of the dot tells us the data status (data available or not), the cloud symbol indicates the connectivity status. The arrow around the dot on the map represents the direction in which the ship is headed.

SymbolMeaning
green_dot_grey_cloud.pngThe latest data for this ship has been uploaded within the last 24 hours, so the ship is considered to be online (green dot). The ship does not have an STS installation (indicated by the gray crossed-out cloud), so we get the data of this ship via other means and have no direct access to our HOMIP device on board.
red_dot_red_cloud.pngThe latest data for this ship is more than 24 hours old, so the ship is considered to be offline (red dot). The ship has an STS installation but is not providing a heartbeat signal, which is indicated by the red cloud.
green_dot_green_cloud.pngThe latest data for this ship has been uploaded within the last 24 hours, so the ship is considered to be online (green dot). The ship has an STS installation and is providing a heartbeat signal, which is indicated by the green cloud.
lucidchart_circle_indicator.pngThere are four ships in this cluster.
The inner circle indicates the online status of the vessels: in this case, two vessels are online (green inner arc) and two vessels are offline (red inner arc). 
The outer circle indicates the heartbeat status of the vessels.
One of the two vessels which are online (green inner arc) has not been providing its heartbeat signal in time (red outer arc) while the other one is (green outer arc). 
Of the two vessels which are offline (red inner arc), one has not been providing its heartbeat signal in time (red outer arc) while the other does not have an STS installation so we do not expect a heartbeat signal (gray outer arc). 

When hovering over individual ships, you can obtain further information, like the ship's name and the last time the ship connected (in case it has STS enabled). An example for this can be seen in the following screenshot:

image2022-6-23_14-32-12.png

If you would like to drill down into the data status of the vessel, a single click is will open the vessels connectivity information. A click on a vessel symbol in the map or in the list view will open up the vessel information card as a drawer on the right-hand side with additional information about the ship. The different regions of the vessel information card are explained in more detail below.

Ship Information‚Äč

The upper part of the drawer provides a general overview of the ship. The ship's name, its IMO number, data connectivity status and a picture of the vessel (if available) will be displayed. 

image2022-6-23_14-33-6.png

STS Device Information‚Äč

If the ship has a HOMIP device on board with a correctly configured STS (Ship  to Shore) installation, this section will display information about the device itself, as well as the installed software packages.

image2022-6-23_14-34-22.png

Nautical Information‚Äč

Nautical information for the selected ship is displayed in this section, which includes details about the nearby port if that information is available.

image2022-6-23_14-34-49.png

Event Notification List‚Äč

If STS version 1.2.1 or later is installed on the HOMIP device, all events displayed on the device itself are transmitted to the shore. These information will be displayed in the following "Notifications" list. Please note that the datasolutions team might consult in terms of updates and upgrades. 

The list always displays the latest state of a notification. In case if the notification was in the alarm state but has since been acknowledged, only the acknowledged status will be shown. Furthermore, the notification list is by default sorted in descending order of event time. The user has the ability to sort any of column in descending or ascending order by clicking on the column header.

image2022-6-23_14-38-44.png

User can also filter content of each column by first hovering over the a column header and then clicking on the three dots that appears which opens up a menu (as shown below).

image2021-11-25_11-47-30.png

image2021-11-25_11-47-13.png

Clicking on the "Filter" option opens up a dialogue box (shown below) from where a user can select the column on which the filter should be applied and the next two options i.e. "Operators" and "Value" would then be adjusted based on the content shown in the selected column.

image2021-11-25_11-48-19.png 

If no events had been transmitted by the HOMIP, or the device does not have STS version 1.2.1 or later installed, the following is shown:

image2021-3-25_14-8-42.png

Furthermore, upon a clicking a single event from the notification list above, a pop-up will open providing a detailed history of the selected event's status over a period of four weeks. The time period can be adjusted via the date time fields provided in the upper part of the pop-up. By this the user has the chance to detect specific event patterns.

image2022-6-23_14-39-39.png

Ship Details‚Äč

At the top right corner of the drawer we have the "Details" button, which when clicked will open up the ship's details page with many functionalities.

image2022-8-29_15-35-9.png

HOMIP Health‚Äč

General Information‚Äč

This part displays some of the already provided information from the drawer. You can find information about the ship itself, as well as all existing notifications of the HOMIP (embedded iPC on board).

Syslog and Tracking Files‚Äč

Besides the mentioned content, if there are system log files or tracking files available, up to the latest two of them are shown here. The files can be issued by the Hoppe Service technician via remote commands from the vessel directly. Often times, these information can provide valuable information for troubleshooting and debugging.

caution

System log files and tracking files contain a lot of specific, technical information and are not intended or adjusted for external use. If you are curious anyway, e.g. to crosscheck the parameter setup of your device, reach out to datasolutions@hoppe-marine.com for further support and advise.

Raw Data Access‚Äč

Transmission Histogram‚Äč

The data upload status is displayed in a swarm plot as can be seen in the following figure.

image2021-6-23_12-9-32.png

While the chosen representation might seem overwhelming at first, you will quickly gain experience in reading the information quickly. The swarm plot provides an immediate overview as to what is happening on the vessel.

If no STS service is available for the ship and we acquire the data by other means, then no STS data will be displayed. Every available dot represents a data file. The horizontal axis is the time axis. The greater the density of points, the more files were transmitted at that time. Additionally, the file size is encoded in the representation: the larger the dot, the larger the file that it represents.

image2021-6-23_12-21-7.png

As can be seen, we structured the representation into two rows. The upper row indicates the time at which the file was created while the lower one indicates the time, the file was transmitted to shore. The following example displays how we can use this information. While it can be seen, that the files where continuously created on the ship (data collection on board is working), we had no transmission till 20.06.2021 and then the transmission recovered and all pending files were transmitted in a batch at around 21.06.2021. After this initial rehydration of the storage, the transmission was restored to normal, and for each created file we see a subsequent transmission event.

image2021-6-23_12-59-54.png

Additionally, hovering over a dot would give you immediate summary of some of the key properties of the file e.g. file creation date, file transmission date and file size.

image2021-6-23_12-37-23.png

CSV Download‚Äč

There are several use cases, where a certain time-range of a ship specific set of signals and with a certain aggregation data is requested in CSV format. In order to fulfill this requirement we have the "CSV Download" option, which looks like this:

image2022-8-26_8-22-7.png

The user can then select multiple signals and provide the desired aggregation and time duration in the respective fields. The user can control the amount of data to be processed in a single attempt:

  • User can select up to 200 signals at a time
  • The time duration can be up to 14 days
  • Configurable aggregation between 600s to 24hrs (86400s)

The "Download CSV" button is enabled as soon as all fields have been filled and have passed the validation checks described above.

image2022-8-26_8-24-40.png

As soon as the user clicks on the "Download CSV" button, the application attempts to fetch the timeseries data for the given parameters and if there is data available then it is downloaded as a CSV file otherwise an error message is shown. 

File Data Access‚Äč

The Fleet Data - Service Portal provides easy access to the data of all managed ships. To access the data head to the "File Data Access" section and proceed as advised by selecting a download format. Clicking the "Download" button will store the file on your local disk in it's source format s3db. This way availability of original database file by iDBS on the ship is ensured. For additional convenience we provide the opportunity to on-the-fly convert s3db files into csv files for easier handling.

image2022-8-26_8-28-13.png

Recent Events‚Äč

Possible STS events which might occur are described in the following list:

Event NameEvent Description
DiskFullDisk usage above 90 percent.
DiskAbove80pcDisk usage above 80 percent.
InboxNotProcessedFiles found in inbox with an age > 1 day.
OutboxTruncatedOut box folder size above configured upper level. Oldest files deleted.
RejectedFolderTruncatedRejected folder truncated since it was too big. Oldest files deleted.
OutboxFileDeleteFile was manually deleted from Outbox by the crew/service technician.
OutboxFileDownloadFile was manually downloaded from Outbox by the crew/service technician.
RejectedFileDeleteRejected file was deleted manually by the crew/service technician.
MonthlyQuotaExceededMonthly transfer quota exceeded.
VeryLowTransferSpeedSatellite link too slow.
RetryCountExceededThe file exceeded the retry count and was put to the rejected folder.
ResumeCountExceededThe file exceeded the resume count and was put to the rejected folder.
ExponentialBackOffIncreasedExponential back off was increased.
ExponentialBackOffResetExponential back off was reset.
Md5MismatchFile transferred is not the same on both sides (shore, vessel) and thus fails md5 checksum.
StsClientSafeModeSTS Client config replaced by Safe mode file.
ConfigUpdateSuccessConfiguration update was success.
ConfigUpdateErrorConfiguration update error.
ConfigUpdateFromRestConfiguration update was successful via the HOWEB interface.
ConfigSafeModeFromRestConfiguration was replaced by the safe-mode file via the HOWEB interface.
FileRejectedTooBigThe file was rejected since it was too big.

Recent Transmissions‚Äč

Last not least the tab "Recent Transmissions" displays the last 50 transmissions which took place for the selected device:

image2022-8-26_8-29-51.png

It is shown in which direction the transmission took place, when it startet and ended as well as the transferred size of the file. The file name tells you of which kind it was, regular data files are displayed as well as software updates and remote commands going from shore side to the vessel.

Export Mappings‚Äč

The backbone of the data transformation is the Hoppe developed python based data transformation tool json-data-transformer. This tool provides means to generate arbitrary JSON output formats from a well defined input data format. This tool can be used to transform the output of Hoppes Time Series Data API into almost arbitrary output structures. Additionally versatile math operations can be performed on the output data to condition the data to customers needs (unit conversion, scaling, offset, rounding, and alike).

The following information about the tool is taken from the readme file of the tool:

The module provides means to transform json input data into a new json output data format which can be defined in a simple descriptive language.

Example mapping‚Äč

Take an data source format, as it is provided via the Hoppe data API:

{
"imo": {
"2020-06-16T15:07:16.022Z": "9461427",
"2020-06-16T15:22:16.011Z": "9461427"
},
"me.fts.act.dgc@avg": {
"2020-06-16T15:07:16.022Z": 37.14391357,
"2020-06-16T15:22:16.011Z": 37.1570279694
},
"timestamp": {
"2020-06-16T15:07:16.022Z": 1592320036.0222222805,
"2020-06-16T15:22:16.011Z": 1592320936.011111021
}
}

Now we have a customer who want's to retrieve the data in e.g. the following format:

[
{
"engine": {
"mainEngines": [
{
"fuelFlows": [
{
"inletTemp": 37.1439135699802,
"type": "HFO"
}
]
}
]
},
"imoNumber": "9461427",
"timestampUtc": "2020-06-16T15:07:16.022222"
},
...
]

It becomes clear, that there is a one to one mapping between the two formats. This mapping is given by

{
"mapping": {
"ti.utc.act.ts@avg": [{
"conversionOperation": {
"arguments": {
"x": "ti.utc.act.ts@avg"
},
"conversion": "fromtimestamp({x})"
},
"targetName": "timestampUtc"
}],
"imo": [{
"targetName": "imoNumber"
}],
"me.fts.act.dgc@avg": [{
"targetName": "engine.mainEngines[0].fuelFlows[0].inletTemp"
},
{
"conversionOperation": {
"conversion": "HFO"
},
"targetName": "engine.mainEngines[0].fuelFlows[0].type"
}]
},
"version": 16,
"time_format": "iso",
"structure_delimiter": ".",
"access_groups": [
"all"
],
"export_format": "nested_json",
"name": "company"
}

This module will take in the input data and the mapping information and return the output format.

With this tool in place we can now request customer data in customer specific formats by providing the export_format=FORMAT parameter to the API query, where FORMAT is the name of the mapping and it's version like in this case e.g. company:9 to provide the data in version 9 of the specific mapping.

These mappings can be conveniently managed by trained Hoppe employers via the Fleet Data - Service Portal available at https://admin.hoppe-sts.com. A screenshot of the management UI which has been developed in the course of this project as well can be seen below:

export_mapping_example.png

caution

In order to create new mappings or update already existing ones, please get in contact with the Data Services Team (datasolutions@hoppe-marine.com).

High Resolution Data‚Äč

In order to retrieve High Resolution Data from the HOMIP device, it is possible to setup a custom request by clicking on the "Create" button on the top right corner. When selecting a ship and device in the overview, if no request was send before, the screen shows no results yet.

image2022-8-22_16-29-8.png

To create a valid request, you have to provide all information shown in the following window:

image2022-8-22_16-31-3.png

Please select a ship, the HOMIP device on the ship, a valid time range, aggregation rate and choice of signals. To finalize your request, click on the "Save" button and confirm the following pop-up, if you are sure:

image2022-8-22_16-35-23.png

The command will then be send to the ship and executed by the HOMIP device. This could take a couple of minutes. The request will appear in the list and keep you up-to-date about the current state:

image2022-8-23_10-54-19.png

While status is "Pending", the command is on its way from shore to ship. As soon as the execution is successful, you see the following changes:

image2022-8-23_10-56-25.png

Clicking on the command itself lead you to an overview page, where you can also see the results of your request. The command interface with all query parameters together with some information is displayed and you can excess an s3db file as payload of the custom export:

image2022-8-23_11-1-44.png

To have a fast look into the payload, there is no download needed. Just click on the "Result Payload Preview" button and a small overview of your request will be displayed. Currently, most files exceed the preview limit, which will lead to the following message:

image2022-8-23_11-20-54.png