Recent questions tagged Hadoop

0 votes

555 views

1 answer

hadoop - How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file ... .i am tried through winscp . See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

358 views

1 answer

hadoop - Confusion with the external tables in hive

I have created the hive external table using below command: use hive2; create external table depTable (depId int ... it doesn't get deleted. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

685 views

1 answer

hadoop - Load data into Hive with custom delimiter

I'm trying to create an internal (managed) table in hive that can store my incremental log data. The ... delimiters and load data successfully. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

512 views

1 answer

hadoop - How to use Sqoop import command with --map-column-hive?

I'm trying to Sqoop the data from Teradata to hive. I thought of following the below steps: 1) Create ... in the corresponding Hive table? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

678 views

1 answer

hadoop - connect to hive in a secured kerberos authenticated cluster using keytab

I am using CDH 5.3.3 and using hive JDBC driver to connect to hive in the secured cluster. I ... jdbc.HiveDriver.connect(HiveDriver.java:104) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

702 views

1 answer

hadoop - How to read gz files in Spark using wholeTextFiles

I have a folder which contains many small .gz files (compressed csv text files). I need to read them in my ... name using sc.textFile(...) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

438 views

1 answer

hadoop - Hive command to execute NOT IN clause

I have two tables,tab1 & tab2. tab1(T1) tab2(T2) a1 b1 b1 c1 c1 f1 d1 g1 I am looking for the values from ... join tab2 on (tab1.T1!=tab2.T2); See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

448 views

1 answer

hadoop - How to write to HDFS using Scala

I am learning Scala and i need to write a custom file to HDFS. I have my own HDFS running on a Cloudera image using ... ) } println("Done!") } } See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

588 views

1 answer

hadoop java.net.URISyntaxException: Relative path in absolute URI: rsrc:hbase-common-0.98.1-hadoop2.jar

I have a map reduce job that connects to HBASE and I can't figure out where I am running into this error ... advance for any help or direction. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

529 views

1 answer

hadoop - Small files and HDFS blocks

Does a block in Hadoop Distributed File System store multiple small files, or a block stores only 1 file? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

553 views

1 answer

hadoop - Connection Error in Apache Pig

I am running Apache Pig .11.1 with Hadoop 2.0.5. Most simple jobs that I run in Pig work perfectly fine. ... how to get rid of these messages? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

496 views

1 answer

hadoop - Namenode file quantity limit

Any one know how many bytes occupy per file in namenode of Hdfs? I want to estimate how many files can store in single namenode of 32G memory. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

451 views

1 answer

hadoop - Where HDFS stores files locally by default?

I am running hadoop with default configuration with one-node cluster, and would like to find where HDFS stores files locally. Any ideas? Thanks. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

577 views

1 answer

hadoop - Delete files older than 10days on HDFS

Is there a way to delete files older than 10 days on HDFS? In Linux I would use: find /path/to/directory/ ... done based on file creation date) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

535 views

1 answer

hadoop - Would Spark unpersist the RDD itself when it realizes it won't be used anymore?

We can persist an RDD into memory and/or disk when we want to use it more than once. However, do we ... myself, I get slower performance. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

657 views

1 answer

hadoop - Append data to existing file in HDFS Java

I'm having trouble to append data to an existing file in HDFS. I want that if the file exists then append a ... I'm missing or doing wrong? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

450 views

1 answer

hadoop - Large Block Size in HDFS! How is the unused space accounted for?

We all know that the block size in HDFS is pretty large (64M or 128M) as compared to the block size in ... please throw some light on this. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

528 views

1 answer

hadoop - How to export data from Spark SQL to CSV

This command works with HiveQL: insert overwrite directory '/data/home.csv' select * from testtable; But with ... CSV feature in Spark SQL. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

482 views

1 answer

hadoop - How to load data to hive from HDFS without removing the source file?

When load data from HDFS to Hive, using LOAD DATA INPATH 'hdfs_file' INTO TABLE tablename; command, it looks ... be used by another process. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

565 views

1 answer

hadoop - Do exit codes and exit statuses mean anything in spark?

I see exit codes and exit statuses all the time when running spark on yarn: Here are a few: CoarseGrainedExecutorBackend ... on a *lost* node See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

688 views

1 answer

hadoop - Are Hive's implicit joins always inner joins?

The join documentation for Hive encourages the use of implicit joins, i.e. SELECT * FROM table1 t1, ... the above return additional records? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

383 views

1 answer

hadoop - Why my BroadcastHashJoin is slower than ShuffledHashJoin in Spark

I execute a join using a javaHiveContext in Spark. The big table is 1,76Gb and has 100 millions record. The ... are stored as Parquet file. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

385 views

1 answer

hadoop - Can brute force algorithms scale?

I have a math problem that I solve by trial and error (I think this is called brute force), and the program ... is more of a general question. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

655 views

1 answer

hadoop - Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?

I have some web server logs that I'd like to query with Hive. The directory structure, in HDFS, looks ... running those 4 commands every time? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

589 views

1 answer

hadoop - Spark Scala list folders in directory

I want to list all folders within a hdfs directory using Scala/Spark. In Hadoop I can do this by using the ... file system with schema file//. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

540 views

1 answer

hadoop - Behavior of the parameter "mapred.min.split.size" in HDFS

The parameter "mapred.min.split.size" changes the size of the block in which the file was written earlier? ... occupy blocks in HDFS 128M; See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

390 views

1 answer

hadoop - Cannot Read a file from HDFS using Spark

I have installed cloudera CDH 5 by using cloudera manager. I can easily do hadoop fs -ls /input/war-and- ... same file by using hadoop commands? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

787 views

1 answer

hadoop - apache spark - check if file exists

I am new to spark and I have a question. I have a two step process in which the first step write a ... . Hoping to find a better alternative. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags