Skip to content

Step by step ingest operation

First step: Create a new media package

  • /ingest/createMediaPackage
    • You will obtain the event-id from the XML response.
    • The xml response is stored in a local variable.

Example:

Bash:

HOST="https://ocweb-prod.cern.ch"
USER="myuser"
PASSWORD="mypassword"
TMP_MP="$(mktemp)"

# Create media package
curl --insecure -f -u ${USER}:${PASSWORD} \
      "${HOST}/ingest/createMediaPackage" -o "${TMP_MP}"

# Retuened mediapackage ID
mediaPackageId=$(head -n 1 ${TMP_MP})
id=$(grep -oPm1 'id="\K[^"]*' <<< "$mediaPackageId")

Python:

#!/usr/bin/python3
import requests
from xml.etree import ElementTree
from datetime import datetime, timedelta
import os, sys
from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor
from requests.auth import HTTPDigestAuth


API_ENDPOINT="https://ocweb-prod.cern.ch/ingest"
USER="myuser"
PASSWORD="mypassword"

session = requests.Session()
session.auth = (USER, PASSWORD)

def create_media_package():
    print("createMediaPackage")
    r = session.get("{0}/createMediaPackage".format(API_ENDPOINT), verify=False)
    tree = ElementTree.fromstring(r.content)
    media_package_id = tree.attrib["id"]
    return media_package_id, r.content

 mp_id, mp_xml = create_media_package()

Response:

<mediapackage xmlns="http://mediapackage.opencastproject.org" 
id="00b0e8c1-7aa5-4010-b277-e0dfd96fd39c" 
start="2021-07-23T08:50:02Z">
<media/><metadata/><attachments/><publications/></mediapackage>

You have to store the returned mediapackage XML into a variable to parse it to the following operations as a form parameter (mediaPackage). You have to update the variable after each operation as it will contain the new added information.

Second step: Add the metadata

Attach the episode metadata file describing the content.

The DATE (temporal) and SERIE (isPartOf) are mandatory.
* /ingest/addDCCatalog

Bash:

START="$(date -d "1 min" --utc +%Y-%m-%dT%H:%MZ)"
END="$(date -d "2 min" --utc +%Y-%m-%dT%H:%MZ)"
SERIES="OPENCAST_SERIES_ID"

echo '<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<dublincore xmlns="http://www.opencastproject.org/xsd/1.0/dublincore/"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <dcterms:creator>taas</dcterms:creator>
  <dcterms:contributor>No contributor</dcterms:contributor>
  <dcterms:created xsi:type="dcterms:W3CDTF">'"${START}"'</dcterms:created>
  <dcterms:temporal xsi:type="dcterms:Period">start='"${START}"'; end='"${END}"'; scheme=W3C-DTF;</dcterms:temporal>
  <dcterms:description>Ingest example</dcterms:description>
  <dcterms:subject>taas</dcterms:subject>
  <dcterms:language>eng</dcterms:language>
  <dcterms:spatial>Remote</dcterms:spatial>
  <dcterms:title>Ingest example</dcterms:title>
  <dcterms:isPartOf>'"${SERIES}"'</dcterms:isPartOf>      
</dublincore>' > "${TMP_DC}"

# Add DC Episode metadata
curl --insecure -f -u ${USER}:${PASSWORD} \
    "${HOST}/ingest/addDCCatalog" -F "mediaPackage=<${TMP_MP}" \
    -F "dublinCore=<${TMP_DC}" \
    -F "flavor=dublincore/episode" -o "${TMP_MP}"

Python:

def add_metadata(media_package_xml):
    now = datetime.utcnow()
    start = now.strftime("%Y-%m-%dT%H:%M")
    end = (now + timedelta(hours=1)).strftime("%Y-%m-%dT%H:%M")
    xml_metadata = """<?xml version="1.0" encoding="UTF-8" ?>
    <dublincore
        xmlns="http://www.opencastproject.org/xsd/1.0/dublincore/"
        xmlns:dcterms="http://purl.org/dc/terms/"
        xmlns:oc="http://www.opencastproject.org/matterhorn/"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <dcterms:creator>taas</dcterms:creator>
            <dcterms:contributor>No contributor</dcterms:contributor>
            <dcterms:created>{start}</dcterms:created>
            <dcterms:temporal xsi:type="dcterms:Period">start={start}; end={end}; scheme=W3C-DTF;</dcterms:temporal>
            <dcterms:description>Ingest example</dcterms:description>
            <dcterms:subject>taas</dcterms:subject>
            <dcterms:language>eng</dcterms:language>
            <dcterms:spatial>Remote</dcterms:spatial>
            <dcterms:title>Ingest example</dcterms:title>
            <dcterms:isPartOf>{series_id}</dcterms:isPartOf>
    </dublincore>
    """
    form_data = dict(
        mediaPackage=media_package_xml,
        flavor="dublincore/episode",
        dublinCore=xml_metadata.format(start=start, end=end, series_id=OPENCAST_SERIES_ID),
    )
    r = session.post("{0}/addDCCatalog".format(API_ENDPOINT), data=form_data, verify=False)

    return r.content

mp_id, mp_xml = create_media_package()
mp_xml=add_metadata(mp_xml)

You will replace the {series_id} by your own assigned series identifier. Remember to store the xml result for the next operation.

Third step: Add the acls

Attach the permissions file that will allow yourself to access the uploaded content.

In this example we will use an external xml file that you can use as a template.

Bash:


TMP_ACL="./files/acl.xml"

# Episode ACL XML
curl -v --insecure -u ${USER}:${PASSWORD} \
  "${HOST}/ingest/addAttachment" -F "mediaPackage=<${TMP_MP}" \
  -F "flavor=security/xacml+episode"  -F "BODY=@${TMP_ACL}" -o "${TMP_MP}"

Python

def add_acl(media_package_xml, acl_filepath):
    print("addAttachment ACL")
    form_data = dict(
        mediaPackage=media_package_xml,
        flavor="security/xacml+episode"
    )
    with open(acl_filepath, "rb") as f:
        print("started streaming acl file")
        files=dict(
            acl=f
        )
        r = session.post("{0}/addAttachment".format(API_ENDPOINT), files=files, data=form_data, verify=False)
        print(r.status_code, r.content)
        print(r.request.url)
        print(r.request.body)
        print(r.request.headers)
        return r.content

acl_filepath = "./files/acl.xml"
mp_id, mp_xml = create_media_package()
mp_xml=add_metadata(mp_xml)
mp_xml=add_acl(mp_xml, acl_filepath)

Fourth step: Add the tracks

  • /ingest/addTrack

You will attach as many video files as you need, but all of them require a "flavour" that will let the Encoding system know how to process them. The two standar flavors are presenter/source and presentation/source for the camera and slides respectively. Anyway, you can use other custom flavours such as video/source or media/source as long as the workflow parses these flavors.

For one single video either presenter or presentation simply omit the other.

Bash:

TMP_PR="./files/camera.mp4"
TMP_SL="./files/slides.mp4"

# Add Presenter/Camera
if [ -f "$TMP_PR" ]; then
    echo "$TMP_PR exists."
    curl --insecure --insecure -f -u ${USER}:${PASSWORD} \
      "${HOST}/ingest/addTrack" -F "flavor=presenter/source" -F "mediaPackage=<${TMP_MP}" \
      -F "BODY=@${TMP_PR}" -o "${TMP_MP}"
      else 
    echo "$TMP_PR does not exist. We do not abort"
fi

# Add Presentation/Slides
if [ -f "$TMP_SL" ]; then
    echo "$TMP_SL exists."
    curl --insecure -f -u ${USER}:${PASSWORD} \
      "${HOST}/ingest/addTrack" -F "flavor=presentation/source" -F "mediaPackage=<${TMP_MP}" \
      -F "BODY=@${TMP_SL}" -o "${TMP_MP}"
      else 
    echo "$TMP_SL does not exist. We do not abort"
fi

Python: For small files you can simply open and send the file.


def add_track(media_package_xml, video_filepath, flavor):
    print("addTrack")
    form_data = dict(
        mediaPackage=media_package_xml,
        flavor=flavor
    )
    with open(video_filepath, 'rb') as f:
        print("started streaming video file")
        files=dict(
            BODY=f
        )
        r = session.post("{0}/addTrack".format(API_ENDPOINT), files=files, data=form_data, verify=False)
        return r.content


presenter_filepath = "./camera.mp4"
presentation_filepath = "./slides.mp4"
acl_filepath = "./acl.xml"
mp_id, mp_xml = create_media_package()
mp_xml=add_metadata(mp_xml)
mp_xml=add_acl(mp_xml, acl_filepath)
mp_xml=add_track(mp_xml, presenter_filepath, "presenter/source")
mp_xml=add_track(mp_xml, presentation_filepath, "presentation/source")

However, we have experience some problems with big files so the recommended way for a Python script is to use MultipartEncoder from requests_toolbelt.

from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor

def add_multipart_track(media_package_xml, video_filepath):
    print("addTrack")
    f = open(video_filepath, 'rb')
    print("started streaming video file")
    m = MultipartEncoder(
        fields={
            'flavor':'presenter/source',
            'mediaPackage':media_package_xml,
            'file':(os.path.basename(video_filepath),f,'application/octet-stream')
        }
    )
    monitor = MultipartEncoderMonitor(m, callback=progress_bar)
    print("Content-Type:%s"%(monitor.content_type))
    r = session.post("{0}/addTrack".format(API_ENDPOINT), data=monitor,headers={'Content-Type': m.content_type}, verify=False, timeout=None)
    print(r.status_code, r.content)
    print(r.request.url)
    print(r.request.headers)
    return r.content

def progress_bar(monitor):
    progress = int(monitor.bytes_read/monitor.len*20)
    sys.stdout.write("\r[{}/{}] bytes |".format(monitor.bytes_read, monitor.len))
    sys.stdout.write("{}>".format("=" * progress))
    sys.stdout.write("{}|".format(" " * (20-progress)))
    sys.stdout.flush()

video_filepath = "./camera.mp4"
acl_filepath = "./acl.xml"
mp_id, mp_xml = create_media_package()
mp_xml=add_metadata(mp_xml)
mp_xml=add_acl(mp_xml, acl_filepath)
mp_xml=add_multipart_track(mp_xml, video_filepath)

Fifth and last step: Close the ingest operation.

Close the mediapackage and run the desired workflow

  • /ingest/ingest

Bash:

WF='cern-cds-videos'

#  Ingest Mediapackage with workflow
curl -v --insecure -u ${USER}:${PASSWORD} \
 "${HOST}/ingest/ingest/${WF}" -F "mediaPackage=<${TMP_MP}" \
  -F "publishToCDS=true" -o "${TMP_MP}"

rm -f "${TMP_MP}" "${TMP_WF}"

Python:


def ingest(media_package_xml):
    form_data = dict(
        mediaPackage=media_package_xml,
    )
    r = session.post("{0}/ingest/cern-cds-videos".format(API_ENDPOINT), data=form_data, verify=False)
    return r.content


video_filepath = "./camera.mp4"
acl_filepath = "./acl.xml"
mp_id, mp_xml = create_media_package()
mp_xml=add_metadata(mp_xml)
mp_xml=add_acl(mp_xml, acl_filepath)
mp_xml=add_multipart_track(mp_xml, video_filepath)
mp_xml=ingest(mp_xml)

When all the assets are uploaded, we run the desired workflow that contains the encoding (and any other) operations, as shown in the cern-cds-videos.xml example * You will obtain also the final XML response with the Workflow identifier:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wf:workflow id="170665" state="INSTANTIATED" 
    xmlns:sec="http://org.opencastproject.security" 
    xmlns:mp="http://mediapackage.opencastproject.org" 
    xmlns:wf="http://workflow.opencastproject.org">
<wf:template>cern-cds-videos</wf:template>
<wf:title>CDS Main Workflow</wf:title>
    ...