topic Re: route to minio in Talend Studio

route to minio

MBourassa1682971203 — Fri, 15 Nov 2024 21:50:21 GMT

Hi,

Sylvain Blanckeart from Talend has developping a route for us who takes parquets file generated from a previous job and put theses files in a minio bucket.

The previous job takes a table from a data base and put it in a csv file maybe more than one for a big table before to convert the files in parquet format in a directory who is monitoring by a route with a cFile.

I have two issuses, the first is if the route is running and I put a parquet file in the directory monitoring by a "cFile", the file who is put in minio is empty but not in the original directory. I dont have this probleme if I run the route after the files are in the directory.

I'm asking me, maybe the file is send to minio before the parquet file is filled ?

The second issus I have is when I deploy this route in a remote engine, I get this error:

I don't get this error in my talend studio.

org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[3DB65D25E6CD4DE-0000000000000240]

at org.apache.camel.CamelExecutionException.wrapCamelExecutionException(CamelExecutionException.java:45) ~[bundleFile:3.11.1]

at org.apache.camel.support.AbstractExchange.setException(AbstractExchange.java:589) ~[bundleFile:3.11.1]

at org.apache.camel.support.DefaultExchange.setException(DefaultExchange.java:27) ~[bundleFile:3.11.1]

at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:69) [bundleFile:3.11.1]

at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:197) [bundleFile:3.11.1]

at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:318) [bundleFile:3.11.1]

at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:182) [bundleFile:3.11.1]

at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:463) [bundleFile:3.11.1]

at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:179) [bundleFile:3.11.1.tesb1]

at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64) [bundleFile:3.11.1.tesb1]

at org.apache.camel.processor.Pipeline.process(Pipeline.java:184) [bundleFile:3.11.1]

at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398) [bundleFile:3.11.1.tesb1]

at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:492) [bundleFile:3.11.1]

at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245) [bundleFile:3.11.1]

at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206) [bundleFile:3.11.1]

at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:190) [bundleFile:3.11.1]

at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:107) [bundleFile:3.11.1]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

at java.lang.Thread.run(Thread.java:829) [?:?]

Caused by: java.lang.NoClassDefFoundError: org/simpleframework/xml/strategy/Strategy

at io.minio.S3Base.createMultipartUpload(S3Base.java:1689) ~[?:?]

at io.minio.S3Base.putObject(S3Base.java:1165) ~[?:?]

at io.minio.MinioClient.putObject(MinioClient.java:1666) ~[?:?]

at org.apache.camel.component.minio.MinioProducer.putObject(MinioProducer.java:171) ~[?:?]

at org.apache.camel.component.minio.MinioProducer.process(MinioProducer.java:79) ~[?:?]

at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66) ~[bundleFile:3.11.1]

... 19 more

Caused by: java.lang.ClassNotFoundException: org.simpleframework.xml.strategy.Strategy cannot be found by wrap_file__E__Talend_8.0.1-R2022-10-RT_runtime_system_io_minio_minio_8.2.2_minio-8.2.2.jar_0.0.0

at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:511) ~[org.eclipse.osgi-3.13.300.jar:?]

at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422) ~[org.eclipse.osgi-3.13.300.jar:?]

at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:414) ~[org.eclipse.osgi-3.13.300.jar:?]

at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:153) ~[org.eclipse.osgi-3.13.300.jar:?]

at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]

at io.minio.S3Base.createMultipartUpload(S3Base.java:1689) ~[?:?]

at io.minio.S3Base.putObject(S3Base.java:1165) ~[?:?]

at io.minio.MinioClient.putObject(MinioClient.java:1666) ~[?:?]

at org.apache.camel.component.minio.MinioProducer.putObject(MinioProducer.java:171) ~[?:?]

at org.apache.camel.component.minio.MinioProducer.process(MinioProducer.java:79) ~[?:?]

at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66) ~[bundleFile:3.11.1]

... 19 more

The route is configured like this:

It begin with a cFile and send to minio with a cJavadslProcessor and there is a cConfig for adding this library: mvn:org.talend.libraries/camel-minio-alldep-3.11.1/6.0.0/jar

the cJavaDSLProcessor code is: .toD("minio://" + context.Ctx_Minio_BucketName + "?accessKey=" + context.Ctx_Minio_AccessKey + "&secretKey=" + context.Ctx_Minio_SecretKey + "&endpoint=" + context.Ctx_Minio_Endpoint + "&region=" + context.Ctx_Minio_Region + "&keyName=${headers.CamelFileName}")

N.B.: My prefered language is french, so, you could response to me in french if you could.

Thank you

Re: route to minio

MBourassa1682971203 — Mon, 08 May 2023 18:16:03 GMT

I just find this error on the remote remote engine log:

Message History (complete message history is disabled)

---------------------------------------------------------------------------------------------------------------------------------------

RouteId ProcessorId Processor Elapsed (ms)

[r000_Main_Fichier_] [r000_Main_Fichier_] [from[file://E:/Talend/data/2_output/minio?autoCreate=true&bufferSize=128&flatt] [ 63]

...

[r000_Main_Fichier_] [toD6 ] [minio://red-test?accessKey=xxx&secretKey=xxxxxx&endpoint=minio.xxx.xxx ] [ 0]

Stacktrace

---------------------------------------------------------------------------------------------------------------------------------------

org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[3DB65D25E6CD4DE-00000000000002B2]