Change in number of lines compiled using ./compile.py in Ephemeral and MP SPDZ

[rohithvaidya]

We were trying to run a .mpc file in ephemeral, which took a long time to compile and then produced an error to the user. It did successfully compile in the background, but the cli produced a "connection reset" error.

But, one thing we observed was a drastic difference in the lines of compilation that ephemeral produced (~9 million lines) as compared to mp spdz. In ephemeral,

When we compiled the same program in MP-SPDZ, we observed that it compiled faster with fewer lines of compilation (0.5 million).

Why is there a discrepancy between these two compilation outputs even though the bit length, protocol and number of parties are same.

PS: We are using latest version of MP-SPDZ

[sbckr]

Hi @rohithvaidya,

Thanks for opening the issue and providing the detailed logs and screenshots.

Let's break down the two points you raised.

1. The Connection reset Error

The java.net.SocketException: Connection reset error strongly suggests that an intermediate component in the network path is terminating the idle connection. The cli itself (via the underlying java-http-client) has no request timeout configured by default and will wait indefinitely, which points to an external cause for the dropped connection.

While the Ephemeral service is busy compiling and running your large program, it doesn't send any data back to the client. This can trigger idle timeouts in various layers of your infrastructure.

• Cloud Load Balancers: Both AWS and Azure load balancers have configurable idle timeouts that default to values like 60 or 300 seconds.

  • AWS Application Load Balancer (ALB): The setting is called Idle timeout. You can find details on how to change it here: AWS Docs - Idle Timeout.
  • Azure Application Gateway: This is controlled by the Request time-out setting in the HTTP settings. See: Azure Docs - Request Time-out.

• Knative Serving: In addition to the external network layers, it is critical to configure the timeout within Knative itself. Knative manages the lifecycle of the service pods and will terminate requests that exceed its configured duration.

  1. Global Configuration: As described in the Carbyne Stack Deployment Guide, you can set a global maximum timeout (max-revision-timeout-seconds) in the config-defaults ConfigMap for Knative.
  2. Service-Specific Configuration (Recommended): The more flexible approach is to set the timeout directly on the Ephemeral service. Using the Carbyne Stack sdk with the Ephemeral Helm chart, you can do this by adding an annotation in helmfile.d/0300.ephemeral.yaml:

  releases:
  - name: {{ requiredEnv "RELEASE_NAME" }}-ephemeral
     chart: carbynestack-oci/ephemeral
     version: "0.3.0"
     values:
     - ephemeral:
       service:
         annotations:
         # Set the timeout to 30 minutes (1800 seconds)
         - "serving.knative.dev/response-timeout-seconds": "1800"

• Istio Service Mesh: As a final point, be aware that the Istio service mesh also has its own timeout settings within its VirtualService resources, which may need adjustment in complex setups.

2. The Compilation Discrepancy

As you pointed out in the issue, you are using the latest version of MP-SPDZ locally, which is likely the source of the discrepancy in compilation time and output size.

Ephemeral currently uses a much older version of MP-SPDZ (commit hash 642d11f). The MP-SPDZ compiler has received significant optimizations since then, which explains the dramatic improvements you see when compiling locally.

Proposed Solution: Pre-compiling the Program

For larger MPC programs, the best practice is to avoid the long on-the-fly compilation step that is triggering the timeout. You can achieve this by building a custom image that contains your pre-compiled program. This also allows you to use a newer version of MP-SPDZ.

Here’s a basic workflow:

1. Create a Custom MP-SPDZ Base Image:

   • Start with the Dockerfile.spdz from the carbynestack/base-images repository: https://github.com/carbynestack/base-images/blob/master/Dockerfile.spdz.
   • Modify it to check out the latest MP-SPDZ version. Build this image and push it to a container registry.
   • Note: You would need to verify the compatibility between your chosen MP-SPDZ version and the Ephemeral runtime environment.

2. Create a Program-Specific Image:

   • In a new Dockerfile, use your custom base image (from step 1) as the FROM source.
   • COPY your MPC source code into the container. To ensure Ephemeral picks it up automatically, name the source file mpc-program.mpc (e.g., COPY my-program.mpc /mp-spdz/Programs/Source/mpc-program.mpc).
   • Use a RUN instruction to compile the program (e.g., RUN ./compile.py Programs/Source/mpc-program.mpc). This will generate /mp-spdz/Programs/Schedules/mpc-program.sch.
   • When Ephemeral starts a job without any source code provided, it defaults to executing this pre-compiled schedule file as seen here: ephemeral/pkg/ephemeral/spdz.go.
   • Heads-up: The container runtime will overwrite this pre-compiled file if a later job execution includes source code. You could make the file read-only within the container to prevent this.

3. Use Your Custom Image in Ephemeral:

   • To tell Ephemeral to use your new program-specific image, you need to edit the build configuration in the .ko.yaml file located in the Ephemeral repository's root directory: https://github.com/carbynestack/ephemeral/blob/master/.ko.yaml.
   • Update the defaultBaseImage to point to your new image.

This approach should significantly speed up the start of your program execution and, by doing so, avoid the network timeouts.

Let us know if this helps or if you have any more questions.