首页 编程语言正文

apache-spark之在 YARN-Cluster 模式下运行时找不到 Hive 表

jackei 阅读：146 2025-06-02 22:19:02 评论：0

我在 HDP 2.3 上有一个 Spark(版本 1.4.1)应用程序。在 YARN-Client 模式下运行时它工作正常。但是，在 YARN-Cluster 模式下运行它时，应用程序找不到我的任何 Hive 表。

我像这样提交申请:

./bin/spark-submit  
    --class com.myCompany.Main  
    --master yarn-cluster  
    --num-executors 3  
    --driver-memory 4g  
    --executor-memory 10g  
    --executor-cores 1  
    --jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar /home/spark/apps/YarnClusterTest.jar   
    --files /etc/hive/conf/hive-site.xml

以下是日志的摘录: 
5/12/02 11:05:13 INFO hive.HiveContext: Initializing execution hive, version 0.13.1 
15/12/02 11:05:14 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 
15/12/02 11:05:14 INFO metastore.ObjectStore: ObjectStore, initialize called 
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 
15/12/02 11:05:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager worker2.xxx.com:34697 with 5.2 GB RAM, BlockManagerId(1, worker2.xxx.com, 34697) 
15/12/02 11:05:16 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 
15/12/02 11:05:16 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "". 
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
15/12/02 11:05:18 INFO metastore.ObjectStore: Initialized ObjectStore 
15/12/02 11:05:19 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added admin role in metastore 
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added public role in metastore 
15/12/02 11:05:19 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 
15/12/02 11:05:19 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 
15/12/02 11:05:19 INFO parse.ParseDriver: Parsing command: SELECT * FROM streamsummary 
15/12/02 11:05:20 INFO parse.ParseDriver: Parse Completed 
15/12/02 11:05:20 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes. 
15/12/02 11:05:20 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=streamsummary 
15/12/02 11:05:20 INFO HiveMetaStore.audit: ugi=spark   ip=unknown-ip-addr  cmd=get_table : db=default tbl=streamsummary     
15/12/02 11:05:20 DEBUG myCompany.Main$: no such table streamsummary; line 1 pos 14

在我的应用程序需要读取或写入 Hive 表的任何时候，我基本上都会遇到相同的“没有这样的表”问题。

在此处发布 hive-site.xml 以供引用。 
  <configuration> 
 
    <property> 
      <name>ambari.hive.db.schema.name</name> 
      <value>hive</value> 
    </property> 
 
    <property> 
      <name>datanucleus.autoCreateSchema</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>datanucleus.cache.level2.type</name> 
      <value>none</value> 
    </property> 
 
    <property> 
      <name>hive.auto.convert.join</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.auto.convert.join.noconditionaltask</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.auto.convert.join.noconditionaltask.size</name> 
      <value>1968526677</value> 
    </property> 
 
    <property> 
      <name>hive.auto.convert.sortmerge.join</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.auto.convert.sortmerge.join.to.mapjoin</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.cbo.enable</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.cli.print.header</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.cluster.delegation.token.store.class</name> 
      <value>org.apache.hadoop.hive.thrift.ZooKeeperTokenStore</value> 
    </property> 
 
    <property> 
      <name>hive.cluster.delegation.token.store.zookeeper.connectString</name> 
      <value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value> 
    </property> 
 
    <property> 
      <name>hive.cluster.delegation.token.store.zookeeper.znode</name> 
      <value>/hive/cluster/delegation</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.abortedtxn.threshold</name> 
      <value>1000</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.check.interval</name> 
      <value>300L</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.delta.num.threshold</name> 
      <value>10</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.delta.pct.threshold</name> 
      <value>0.1f</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.initiator.on</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.worker.threads</name> 
      <value>0</value> 
    </property> 
 
    <property> 
      <name>hive.compactor.worker.timeout</name> 
      <value>86400L</value> 
    </property> 
 
    <property> 
      <name>hive.compute.query.using.stats</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.conf.restricted.list</name> 
      <value>hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role</value> 
    </property> 
 
    <property> 
      <name>hive.convert.join.bucket.mapjoin.tez</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.default.fileformat</name> 
      <value>TextFile</value> 
    </property> 
 
    <property> 
      <name>hive.default.fileformat.managed</name> 
      <value>TextFile</value> 
    </property> 
 
    <property> 
      <name>hive.enforce.bucketing</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.enforce.sorting</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.enforce.sortmergebucketmapjoin</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.exec.compress.intermediate</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.exec.compress.output</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.exec.dynamic.partition</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.exec.dynamic.partition.mode</name> 
      <value>nonstrict</value> 
    </property> 
 
    <property> 
      <name>hive.exec.failure.hooks</name> 
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value> 
    </property> 
 
    <property> 
      <name>hive.exec.max.created.files</name> 
      <value>100000</value> 
    </property> 
 
    <property> 
      <name>hive.exec.max.dynamic.partitions</name> 
      <value>5000</value> 
    </property> 
 
    <property> 
      <name>hive.exec.max.dynamic.partitions.pernode</name> 
      <value>2000</value> 
    </property> 
 
    <property> 
      <name>hive.exec.orc.compression.strategy</name> 
      <value>SPEED</value> 
    </property> 
 
    <property> 
      <name>hive.exec.orc.default.compress</name> 
      <value>ZLIB</value> 
    </property> 
 
    <property> 
      <name>hive.exec.orc.default.stripe.size</name> 
      <value>67108864</value> 
    </property> 
 
    <property> 
      <name>hive.exec.orc.encoding.strategy</name> 
      <value>SPEED</value> 
    </property> 
 
    <property> 
      <name>hive.exec.parallel</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.exec.parallel.thread.number</name> 
      <value>8</value> 
    </property> 
 
    <property> 
      <name>hive.exec.post.hooks</name> 
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value> 
    </property> 
 
    <property> 
      <name>hive.exec.pre.hooks</name> 
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value> 
    </property> 
 
    <property> 
      <name>hive.exec.reducers.bytes.per.reducer</name> 
      <value>67108864</value> 
    </property> 
 
    <property> 
      <name>hive.exec.reducers.max</name> 
      <value>1009</value> 
    </property> 
 
    <property> 
      <name>hive.exec.scratchdir</name> 
      <value>/tmp/hive</value> 
    </property> 
 
    <property> 
      <name>hive.exec.submit.local.task.via.child</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.exec.submitviachild</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.execution.engine</name> 
      <value>tez</value> 
    </property> 
 
    <property> 
      <name>hive.fetch.task.aggr</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.fetch.task.conversion</name> 
      <value>more</value> 
    </property> 
 
    <property> 
      <name>hive.fetch.task.conversion.threshold</name> 
      <value>1073741824</value> 
    </property> 
 
    <property> 
      <name>hive.limit.optimize.enable</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.limit.pushdown.memory.usage</name> 
      <value>0.04</value> 
    </property> 
 
    <property> 
      <name>hive.map.aggr</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.map.aggr.hash.force.flush.memory.threshold</name> 
      <value>0.9</value> 
    </property> 
 
    <property> 
      <name>hive.map.aggr.hash.min.reduction</name> 
      <value>0.5</value> 
    </property> 
 
    <property> 
      <name>hive.map.aggr.hash.percentmemory</name> 
      <value>0.5</value> 
    </property> 
 
    <property> 
      <name>hive.mapjoin.bucket.cache.size</name> 
      <value>10000</value> 
    </property> 
 
    <property> 
      <name>hive.mapjoin.optimized.hashtable</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.mapred.reduce.tasks.speculative.execution</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.merge.mapfiles</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.merge.mapredfiles</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.merge.orcfile.stripe.level</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.merge.rcfile.block.level</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.merge.size.per.task</name> 
      <value>256000000</value> 
    </property> 
 
    <property> 
      <name>hive.merge.smallfiles.avgsize</name> 
      <value>16000000</value> 
    </property> 
 
    <property> 
      <name>hive.merge.tezfiles</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.authorization.storage.checks</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.cache.pinobjtypes</name> 
      <value>Table,Database,Type,FieldSchema,Order</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.client.connect.retry.delay</name> 
      <value>5s</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.client.socket.timeout</name> 
      <value>1800s</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.connect.retries</name> 
      <value>24</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.execute.setugi</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.failure.retries</name> 
      <value>24</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.kerberos.keytab.file</name> 
      <value>/etc/security/keytabs/hive.service.keytab</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.kerberos.principal</name> 
      <value>hive/_HOST@EXAMPLE.COM</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.pre.event.listeners</name> 
      <value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.sasl.enabled</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.server.max.threads</name> 
      <value>100000</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.uris</name> 
      <value>thrift://host.xxx.com:9083</value> 
    </property> 
 
    <property> 
      <name>hive.metastore.warehouse.dir</name> 
      <value>/apps/hive/warehouse</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.bucketmapjoin</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.bucketmapjoin.sortedmerge</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.constant.propagation</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.index.filter</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.metadataonly</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.null.scan</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.reducededuplication</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.reducededuplication.min.reducer</name> 
      <value>4</value> 
    </property> 
 
    <property> 
      <name>hive.optimize.sort.dynamic.partition</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.orc.compute.splits.num.threads</name> 
      <value>10</value> 
    </property> 
 
    <property> 
      <name>hive.orc.splits.include.file.footer</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.prewarm.enabled</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.prewarm.numcontainers</name> 
      <value>3</value> 
    </property> 
 
    <property> 
      <name>hive.security.authenticator.manager</name> 
      <value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value> 
    </property> 
 
    <property> 
      <name>hive.security.authorization.enabled</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.security.authorization.manager</name> 
      <value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory</value> 
    </property> 
 
    <property> 
      <name>hive.security.metastore.authenticator.manager</name> 
      <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value> 
    </property> 
 
    <property> 
      <name>hive.security.metastore.authorization.auth.reads</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.security.metastore.authorization.manager</name> 
      <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value> 
    </property> 
 
    <property> 
      <name>hive.server2.allow.user.substitution</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.server2.authentication</name> 
      <value>NONE</value> 
    </property> 
 
    <property> 
      <name>hive.server2.authentication.spnego.keytab</name> 
      <value>HTTP/_HOST@EXAMPLE.COM</value> 
    </property> 
 
    <property> 
      <name>hive.server2.authentication.spnego.principal</name> 
      <value>/etc/security/keytabs/spnego.service.keytab</value> 
    </property> 
 
    <property> 
      <name>hive.server2.enable.doAs</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.server2.logging.operation.enabled</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.server2.logging.operation.log.location</name> 
      <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value> 
    </property> 
 
    <property> 
      <name>hive.server2.support.dynamic.service.discovery</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.server2.table.type.mapping</name> 
      <value>CLASSIC</value> 
    </property> 
 
    <property> 
      <name>hive.server2.tez.default.queues</name> 
      <value>default</value> 
    </property> 
 
    <property> 
      <name>hive.server2.tez.initialize.default.sessions</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.server2.tez.sessions.per.default.queue</name> 
      <value>1</value> 
    </property> 
 
    <property> 
      <name>hive.server2.thrift.http.path</name> 
      <value>cliservice</value> 
    </property> 
 
    <property> 
      <name>hive.server2.thrift.http.port</name> 
      <value>10001</value> 
    </property> 
 
    <property> 
      <name>hive.server2.thrift.max.worker.threads</name> 
      <value>500</value> 
    </property> 
 
    <property> 
      <name>hive.server2.thrift.port</name> 
      <value>10000</value> 
    </property> 
 
    <property> 
      <name>hive.server2.thrift.sasl.qop</name> 
      <value>auth</value> 
    </property> 
 
    <property> 
      <name>hive.server2.transport.mode</name> 
      <value>binary</value> 
    </property> 
 
    <property> 
      <name>hive.server2.use.SSL</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.server2.zookeeper.namespace</name> 
      <value>hiveserver2</value> 
    </property> 
 
    <property> 
      <name>hive.smbjoin.cache.rows</name> 
      <value>10000</value> 
    </property> 
 
    <property> 
      <name>hive.stats.autogather</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.stats.dbclass</name> 
      <value>fs</value> 
    </property> 
 
    <property> 
      <name>hive.stats.fetch.column.stats</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.stats.fetch.partition.stats</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.support.concurrency</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.tez.auto.reducer.parallelism</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.tez.container.size</name> 
      <value>5632</value> 
    </property> 
 
    <property> 
      <name>hive.tez.cpu.vcores</name> 
      <value>-1</value> 
    </property> 
 
    <property> 
      <name>hive.tez.dynamic.partition.pruning</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.tez.dynamic.partition.pruning.max.data.size</name> 
      <value>104857600</value> 
    </property> 
 
    <property> 
      <name>hive.tez.dynamic.partition.pruning.max.event.size</name> 
      <value>1048576</value> 
    </property> 
 
    <property> 
      <name>hive.tez.input.format</name> 
      <value>org.apache.hadoop.hive.ql.io.HiveInputFormat</value> 
    </property> 
 
    <property> 
      <name>hive.tez.java.opts</name> 
      <value>-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps</value> 
    </property> 
 
    <property> 
      <name>hive.tez.log.level</name> 
      <value>INFO</value> 
    </property> 
 
    <property> 
      <name>hive.tez.max.partition.factor</name> 
      <value>2.0</value> 
    </property> 
 
    <property> 
      <name>hive.tez.min.partition.factor</name> 
      <value>0.25</value> 
    </property> 
 
    <property> 
      <name>hive.tez.smb.number.waves</name> 
      <value>0.5</value> 
    </property> 
 
    <property> 
      <name>hive.txn.manager</name> 
      <value>org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager</value> 
    </property> 
 
    <property> 
      <name>hive.txn.max.open.batch</name> 
      <value>1000</value> 
    </property> 
 
    <property> 
      <name>hive.txn.timeout</name> 
      <value>300</value> 
    </property> 
 
    <property> 
      <name>hive.user.install.directory</name> 
      <value>/user/</value> 
    </property> 
 
    <property> 
      <name>hive.vectorized.execution.enabled</name> 
      <value>true</value> 
    </property> 
 
    <property> 
      <name>hive.vectorized.execution.reduce.enabled</name> 
      <value>false</value> 
    </property> 
 
    <property> 
      <name>hive.vectorized.groupby.checkinterval</name> 
      <value>4096</value> 
    </property> 
 
    <property> 
      <name>hive.vectorized.groupby.flush.percent</name> 
      <value>0.1</value> 
    </property> 
 
    <property> 
      <name>hive.vectorized.groupby.maxentries</name> 
      <value>100000</value> 
    </property> 
 
    <property> 
      <name>hive.zookeeper.client.port</name> 
      <value>2181</value> 
    </property> 
 
    <property> 
      <name>hive.zookeeper.namespace</name> 
      <value>hive_zookeeper_namespace</value> 
    </property> 
 
    <property> 
      <name>hive.zookeeper.quorum</name> 
      <value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value> 
    </property> 
 
    <property> 
      <name>javax.jdo.option.ConnectionDriverName</name> 
      <value>com.mysql.jdbc.Driver</value> 
    </property> 
 
    <property> 
      <name>javax.jdo.option.ConnectionPassword</name> 
      <value>hive</value> 
    </property> 
 
    <property> 
      <name>javax.jdo.option.ConnectionURL</name> 
      <value>jdbc:mysql://host.xxx.com/hive?createDatabaseIfNotExist=true</value> 
    </property> 
 
    <property> 
      <name>javax.jdo.option.ConnectionUserName</name> 
      <value>hive</value> 
    </property> 
 
  </configuration>

请您参考如下方法:

我在 Hortonworks 社区上发布了同样的问题，并在 this 答案的帮助下解决了这个问题。

其要点是:提交申请时，--files 参数必须在 --jars 参数之前，并且要使用的 hive-site.xml 副本是 Spark conf 目录中的副本，而不是 $HIVE_HOME/conf/hive-site.xml 。因此:

  ./bin/spark-submit \ 
  --class com.myCompany.Main \ 
  --master yarn-cluster \ 
  --num-executors 3 \ 
  --driver-memory 1g \ 
  --executor-memory 11g \ 
  --executor-cores 1 \ 
  --files /usr/hdp/current/spark-client/conf/hive-site.xml \ 
  --jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar \ 
  /home/spark/apps/YarnClusterTest.jar


标签:Spark
声明

1.本站遵循行业规范，任何转载的稿件都会明确标注作者和来源；2.本站的原创文章，请转载时务必注明文章作者和来源，不尊重原创的行为我们将追究责任；3.作者投稿可能会经我们编辑修改或补充。

相关文章
关注我们

一个IT知识分享的公众号

友情链接IM LINKS