hadoop - Presto failing to query hive table -

-

on emr created dataset in parquet using spark , storing on s3. able create external table , query using hive when try perform same query using presto obtain error (the part referred changes @ every run).

2016-11-13t13:11:15.165z        error   remote-task-callback-36 com.facebook.presto.execution.stagestatemachine stage 20161113_131114_00004_yp8y5.1 failed com.facebook.presto.spi.prestoexception: error opening hive split s3://my_bucket/my_table/part-r-00013-b17b4495-f407-49e0-9d15-41bb0b68c605.snappy.parquet (offset=1100508800, length=68781800): null         @ com.facebook.presto.hive.parquet.parquethiverecordcursor.createparquetrecordreader(parquethiverecordcursor.java:475)     @ com.facebook.presto.hive.parquet.parquethiverecordcursor.<init>(parquethiverecordcursor.java:247)     @ com.facebook.presto.hive.parquet.parquetrecordcursorprovider.createhiverecordcursor(parquetrecordcursorprovider.java:96)     @ com.facebook.presto.hive.hivepagesourceprovider.gethiverecordcursor(hivepagesourceprovider.java:129)     @ com.facebook.presto.hive.hivepagesourceprovider.createpagesource(hivepagesourceprovider.java:107)     @ com.facebook.presto.spi.connector.classloader.classloadersafeconnectorpagesourceprovider.createpagesource(classloadersafeconnectorpagesourceprovider.java:44)     @ com.facebook.presto.split.pagesourcemanager.createpagesource(pagesourcemanager.java:48)     @ com.facebook.presto.operator.tablescanoperator.createsourceifnecessary(tablescanoperator.java:268)     @ com.facebook.presto.operator.tablescanoperator.isfinished(tablescanoperator.java:210)     @ com.facebook.presto.operator.driver.processinternal(driver.java:375)     @ com.facebook.presto.operator.driver.processfor(driver.java:301)     @ com.facebook.presto.execution.sqltaskexecution$driversplitrunner.processfor(sqltaskexecution.java:622)     @ com.facebook.presto.execution.taskexecutor$prioritizedsplitrunner.process(taskexecutor.java:529)     @ com.facebook.presto.execution.taskexecutor$runner.run(taskexecutor.java:665)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)     @ java.lang.thread.run(thread.java:745) caused by: java.io.eofexception     @ java.io.datainputstream.readfully(datainputstream.java:197)     @ java.io.datainputstream.readfully(datainputstream.java:169)     @ parquet.hadoop.parquetfilereader.readfooter(parquetfilereader.java:420)     @ parquet.hadoop.parquetfilereader.readfooter(parquetfilereader.java:385)     @ com.facebook.presto.hive.parquet.parquethiverecordcursor.lambda$createparquetrecordreader$0(parquethiverecordcursor.java:416)     @ com.facebook.presto.hive.authentication.nohdfsauthentication.doas(nohdfsauthentication.java:23)     @ com.facebook.presto.hive.hdfsenvironment.doas(hdfsenvironment.java:76)     @ com.facebook.presto.hive.parquet.parquethiverecordcursor.createparquetrecordreader(parquethiverecordcursor.java:416)     ... 16 more

the parquet location constituted 128 parts - data stored on s3 , encrypted using client-side encryption kms. presto uses custom encryption-materials provider (specified using presto.s3.encryption-materials-provider) returns kmsencryptionmaterials object initialized master key. using emr 5.1.0 (hive 2.1.0, spark 2.0.1, presto 0.152.3).

does surface when encryption turned off?

there bugreport surfaced against asf s3a client (not emr one), things breaking when filesystem listed length != actual file length. is: because of encryption, file length in list > length in read.

we couldn't repro in our tests, , our conclusion anyway "filesystems must not that" (indeed, it's fundamental requirement of hadoop fs spec: listed len must equal actual length). if emr code getting wrong, it's in driver downstream code cannot expected handle


Comments

Post a Comment

Popular posts from this blog

php - How to display all orders for a single product showing the most recent first? Woocommerce -

-
i have password protected page on frontend. on page, want display full list of purchases associated single woocommerce product id. option 1: i've looked through woocommerce docs , , found: wc_cli_order , list_( $args, $assoc_args ) function lists orders. presumably, orders can filtered product id [--=] filter orders based on order property. line item fields (numeric array, started index zero): line_items.0.product_id option 2: i found article , based on customer id. i've modified code based on guesses meta_key , meta_value. $customer_orders = get_posts( array( 'numberposts' => -1, 'meta_key' => '_product', 'meta_value' => get_product_id(), 'post_type' => wc_get_order_types(), 'post_status' => array_keys( wc_get_order_statuses() ), ) ); how display orders single product showing recent order first? here solution put based on these articles: http://www.w...
Read more

asp.net - How to correctly use QUERY_STRING in ISAPI rewrite? -

-
i'm having difficult time , hope guys can help. using helicon isapi rewrite version 3.1.0.104 on our iis server. edit http.conf file , have tried , still fail. this trying do: redirect url: https://www.domain.com/switch-by-version?version=2.8.5.2594 to: https://test.domain.com/load/load.aspx?tver=2.8.5.2594 the version number @ end source url change, , need target url have same version number @ end of url example above shows. i tried following not work: rewritecond %{query_string} ^version=(\d\d?)\.(\d\d?)\.(\d\d?)\.(.*)$ [nc] rewriterule ^/switch-by-version(.*)$ https://test.domain.com/load/load.aspx?tver=%1 [r=307,nc,l] any appreciated! thanks. if it's running in equivalent context of .htaccess , rule shouldn't start / . here's simpler version: rewritecond %{query_string} ^version=(\d+\.\d+\.\d+(?:\..+)?)$ [nc] rewriterule ^switch-by-version/?$ https://test.domain.com/load/load.aspx?tver=%1 [r=307,nc,l]
Read more

angularjs - How restrict admin panel using in backend laravel and admin panel on angular? -

-
https://scotch.io/tutorials/token-based-authentication-for-angularjs-and-laravel-apps following tutorials, try write authentication restrict admin panel, didn't understood how can that. this admin panel, admin panel don't want show , request database unauthenticated users. my laravel-routes route::get('/admin',function(){ return view('index'); }); route::get('/api/v1/employees/{id?}', 'employees@index'); route::get('/api/v1/users/{id?}', 'employees@users'); route::post('/api/v1/employees', 'employees@store'); route::post('/api/v1/employees/{id}', 'employees@update'); route::delete('/api/v1/employees/{id}', 'employees@destroy'); route::group(['prefix' => 'api'], function() { route::resource('authenticate', 'authenticatecontroller', ['only' => ['index']]); route::post('authenticate', 'authentic...
Read more